DEV Community

Cover image for Migrations Of Madness
mmcclelland1002
mmcclelland1002

Posted on

Migrations Of Madness

After working with Django for any extended period of time, migrations (and the madness that is often associated with them) are the biggest problem that Djangonauts most commonly encounter. They are simultaneously Django’s most powerful and most misunderstood feature. While they are designed to be mostly automatic, you’ll need to know when to make them, when to run them, and how to handle very common problems when they occur - from an inconsistent migration history, to conflicting migration dependencies, to keeping your migrations synced across all databases and environments. There is a multiverse of issues that could arise.

Let’s dive a little deeper into this madness… How does Django detect model changes and create migrations?

The official Django documentation summarizes this best:

The operations are the key; they are a set of declarative instructions which tell Django what schema changes need to be made. Django scans them and builds an in-memory representation of all of the schema changes to all apps, and uses this to generate the SQL which makes the schema changes.
That in-memory structure is also used to work out what the differences are between your models and the current state of your migrations; Django runs through all the changes, in order, on an in-memory set of models to come up with the state of your models last time you ran makemigrations. It then uses these models to compare against the ones in your models.py files to work out what you have changed.

Django cannot resolve the migration dependencies consistentl
One issue with the makemigrations command is that when creating this in-memory project state, it sometimes cannot resolve the migration dependencies consistently when new migrations (and thus new dependencies) are introduced. This may cause the dependency tree to be rearranged even if the migrations were already applied on the target database. And when this occurs, the dreaded InconsistentMigrationHistory exception is thrown.

Production database like 100 Dr. Stranges casting Eldritch Whips vs. Thanos
If you are not careful, this scenario can lock up your database like a 100 Dr. Stranges casting Eldritch Whips vs. Thanos.

This happens regularly in larger, complex Django projects where large dev teams might be working with a number of cross-app dependencies. We experienced this often at our company when there was rapid development happening with lots of database migrations in a short period of time. Pinpointing the specific migration file that was causing issues was time-consuming and often a trial and error endeavor.

We explored several debugging tools out there (most notably django-extensions and their graph_models management command), but none fit our use case and all lacked several key features that we required:

  1. Easily pluggable and fully automated
  2. Saving migration snapshots to our database in our desired output format
  3. Viewing the migration history based on the migration’s applied time

In comes django-migration-snapshots, which allows us to visualize Django’s migration dependency tree as a directed graph (digraph for short) - along with the above features and more. The package is easily pluggable into any project and provides the ability to easily save a snapshot of our project’s migration history. And since this package is built on top of pygraphviz, the textual and graphical snapshot can be saved in 20+ output formats.

Graphical Snapshot

Graphical output of django-migration-snapshots
Text Snapshot

digraph {
    "admin/0001_initial" -> "auth/0001_initial"
    "admin/0001_initial" -> "contenttypes/0001_initial"
    "admin/0002_logentry_remove_auto_add" -> "admin/0001_initial"
    "admin/0003_logentry_add_action_flag_choices" -> "admin/0002_logentry_remove_auto_add"
    "auth/0001_initial" -> "contenttypes/0001_initial"
    "auth/0002_alter_permission_name_max_length" -> "auth/0001_initial"
    ...
}
Enter fullscreen mode Exit fullscreen mode

Here are the following ways of creating a snapshot:
1) Execute management command to create snapshot
creates snapshot of entire migration history

# creates snapshot of entire migration history
python manage.py create_snapshot

# filter migrations before applied date (YYYY-MM-DD)
python manage.py create_snapshot --date="2022-10-15"
Enter fullscreen mode Exit fullscreen mode

django-migration-snapshots allow you to go back to a certain point in time
Wouldn’t it be nice to turn back time like Dr. Strange with the Time stone and see a snapshot of your migration dependency graph at any given point in time? django-migration-snapshots overrides Django’s built-in MigrationLoader class to allow for this ability.

2) Create object programmatically or from the admin panel

MigrationSnapshot.objects.create(output_format="pdf")
Enter fullscreen mode Exit fullscreen mode

3) Automatically create migration snapshots with the post_migrate signal

from django.apps import AppConfig
from django.db.models.signals import post_migrate

def my_snapshot_callback(sender, **kwargs):
    # Create migration snapshot
    MigrationSnapshot.objects.create(output_format="pdf")

class MyAppConfig(AppConfig):

    def ready(self):
        # send signal only once after all migrations execute
        post_migrate.connect(my_snapshot_callback, sender=self)

Enter fullscreen mode Exit fullscreen mode

While this solution was helpful in our use case, it is most likely unnecessary in smaller teams and smaller django projects where the migration history is relatively straight forward. In addition, regularly squashing migrations is a good practice to follow and will help simplify your migration history over the long run.

This project is still in its infancy and is being developed. All contributions, PR’s, and issues are welcomed: GitHub Repo

Looking for a Django job? Summit Technology Group is a rapidly growing fintech company and we want to hire expert Djangonauts. Please reach out!

Relevant Links

Top comments (0)