Cleaning Up Your Django repo — A Holistic Approach to Managing Migration Files

Implementing the “django-migration-zero” Pattern, Including the CI/CD Pipeline

Published in

ambient-digital

5 min readDec 6, 2023

TL;DR

Your Django migrations are piling up in your repo? You want to clean them up without a hassle? You need a neat solution to make your production database understand that you’ve reset your migration files?

Check out this package: https://pypi.org/project/django-migration-zero/

Install django-migration-zero and deploy changes to all environments (test, stage, prod)
Add manage.py handle_migration_zero_reset to your migration/post-deploy script
Remove and recreate local migration files with manage.py reset_local_migration_files
Commit local changes to a new branch
Go to Django admin and set flag and timestamp
Run deployment
Read the docs properly before starting any of this! ☝️

In a nutshell

The article introduces a solution, django-migration-zero, aimed at efficiently managing Django migration files and cleaning up local migrations. Furthermore, it helps you update your migration history on your environments (like test or production systems).

Why do I need this package?

Migrations will pile up in your application over the course of time. They will clutter your repo, slow down your tests (time is after all money), and can lead to some nasty problems when you merge branches.

Squashing takes time to resolve all those circular dependencies within your models and in the end, you do work for something that you don’t need. All migrations have been applied, so the structural and data changes are not relevant for you anymore.

The “migration zero” approach will help you get rid of those migration files. But most solutions or packages will only help you do a local clean-up, while the tricky part is to update your migration history on your deployed systems.

Certainly, you can log into your production system and run a series of commands manually but this is always a high-stress situation. What if something goes sideways? Secondly, not everybody has access to all environments. And lastly, this task is tedious and takes time. And we all know that you won’t implement a habit (like cleaning up your migrations regularly) when it’s annoying and time-consuming.

This is why we’ve built this package. It helps you clean up all those files AND will handle all the things that need to happen on your databases, no matter where they live and how you can access them.

Getting things cleaned up

Prerequisites

Ensure that your deployment branches are properly merged into each other. This means all commits from master are in stage, all commits from stage are in develop. Your branches might have other names but the important thing is that you don’t have any commits lying around in upstream branches which are not yet inside the downstream ones.

Also ensure that all migrations in the branch you are working on have been applied in its database.

Setup

Install the package django-migration-zero. First, you have to tell the package where your local Django apps live. Usually, they all sit within one directory, in this case called “apps”.

# settings.py
MIGRATION_ZERO_APPS_DIR = BASE_DIR / "apps"

Now, deploy this setup to all of your environments like test, stage, and production. This is very important! Do not continue until this has been successfully deployed. Otherwise, you can’t “prepare” the migration reset release in the Django admin and your next migration run will crash.

Cleaning up local files

Next, create a new git branch. You can now use a handy management command to delete all migration files and recreate them using the well-known makemigrations command.

python manage.py reset_local_migration_files

In case you want to double-check the changes, just use git status.

Updating the CI/CD

Most projects nowadays use a continuous integration/continuous delivery pipeline. This pipeline will automatically ensure quality and handles the deployment to any of your systems.

At some point in your pipeline, you have to run python manage.py migrate. Add this management command BEFORE you migrate:

python manage.py handle_migration_zero_reset

Preparing the target system

Log into your Django admin, look for the “Migration Zero Configuration,” enable the switch, and change the date to the date of the deployment (probably “today”). This step is crucial! Don’t forget it! If you don’t do this, your migration reset deployment won’t do any database clean-up and your next migration run will fail.

Deployment

Merge your merge/pull request and deploy to your target system. This will then execute manage.py handle_migration_zero via your CI/CD and adjust Django’s migration history to reflect the new initial migrations.

Under the hood

If you are curious how the inner workings of this look like — I don’t want to disappoint you.

Clean-up script

The crucial point is the settings variable pointing to the directory of your local Django apps. Since all migration files must live inside an app on the first level inside a /migrations directory, the clean-up script is quite straight forward: Iterate all directories and delete all Python files matching the pattern. Afterward, run makemigrations to create new initial migration files.

CI/CD script

The part for the CI/CD pipeline is a tiny bit trickier, so we have to be a little creative.

We start by checking if the migration configuration singleton was set to “True” via the Django admin. This is the indicator that we are about to push a migration-zero-commit and need to prepare the database accordingly.

The only thing we really must take care of is Djangos history table in the database. This contains our local (to-be-cleaned-up) migrations but the ones from all other third-party apps as well. So, a simple TRUNCATE won’t work.

The script will — nevertheless — empty the table django_migrations and repopulate it with the migrate --fake command. Though this will set the timestamp of the third-party migrations to the current time, it doesn’t have any impact on our database otherwise.

When the table has been repopulated, we check that everything worked fine by running migrate . If the execution works flawlessly, we can be sure that we didn’t break anything.

Finally, we are setting the migration configuration singleton flag back to “False”, so the developer doesn’t have to worry about it.

Conclusion

There are many ways to clean your migrations but in the end, they tend to be tedious and therefore not practical. This approach might feel a little bulky on first installation but once you are up and running, it’s literally just calling a clean-up command, switching the Django admin flag, and deploy your changes.

In my opinion, the biggest USP of this package is the CI/CD support which most custom solutions lack.

I’d appreciate some try-outs and feedback from the Django community! 🐎