Introducing django-linear-migrations

Update (2021-01-07): My new book Boost Your Django DX covers django-linear-migrations and other ways to safeguard your migrations.

If you’ve used Django migrations for a while, you may be familiar with this message:

$ python manage.py migrate
CommandError: Conflicting migrations detected; multiple leaf nodes in the migration graph: (0002_longer_titles, 0002_author_nicknames).
To fix them run 'python manage.py makemigrations --merge'

This appears when the migration history for one of your apps branched to have two “leaf nodes”, that is, two final migrations. The simplest example has our first initial migration, then two conflicting second migrations:

                  +--> 0002_author_nicknames
                 /
0001_initial +--|
                 \
                  +--> 0002_longer_titles

This happens quite naturally when developing two features for same app, both with migrations.

The solution Django suggests is to create a merge migration with makemigrations --merge. This creates another migration in our history that depends on the last two:

                  +--> 0002_author_nicknames +-+
                 /                               \
0001_initial +--|                                 |--> 0003_merge
                 \                               /
                  +--> 0002_longer_titles +----+

This merge migration tells Django “it’s fine to run both branches of migrations and end up here”. This is a simple solution, and avoids modification of the exisiting migrations. But it has a number of drawbacks.

First, it’s a fix after the fact. You need to encounter the “Conflicting migrations detected” error before you step in and create the merge migration. This is normally after merging to your main branch, so requiring your CI to break and wait for a fix. In even a small team with a reasonable pace of change, you might find the CI on your main branch breaking with conflicting migrations several times a week.

Update (2020-12-17): Thanks to Shai Berger for correcting the below section, which previously claimed Django used an alphabetical migration order.

Second, it doesn’t enforce the same execution order in all environments. The generated merge migration may define a different execution order of the migrations to the order that they were committed. When you use merge migrations, you allow two migrations to share the same number, so their order is dependent on the remainder of their names. For example:

1. We merge 0002_longer_titles and our CI automatically runs it it on our staging environment.
2. We merge 0002_author_nicknames, and staging triggers the “Conflicting migrations detected” error.
3. We create a merge migration, 0003_merge, with makemigrations --merge, and this depends on [0002_author_nicknames, 0002_longer_titles] - the opposite order to the merge order.
4. We merge 0003_merge and then 0002_author_nicknames and 0003_longer_titles execute on staging.
5. We deploy to production, which runs the migrations based on the order in 0003_merge: 0002_author_nicknames, 0002_longer_titles, 0003_merge. This is different to the order they executed on staging.

This difference makes our staging environment a less useful simulation of production.

Third, it complicates rollbacks. It’s hard to reverse migrations when you can’t be sure of the order they ran in.

Enter my new package, django-linear-migrations. It ensures the migration history in your apps remains linear, such as:

0001_initial +--> 0002_author_nicknames +--> 0002_longer_titles

It does this by creating new files in your apps’ migration directories, called max_migration.txt. These files contain the latest migration name for that app, and are automatically updated by makemigrations. They mean that if a new migration is merged for an app, any other branches adding migrations for that app will have a merge conflict in your source control tool (e.g. Git).

This forces you to order migrations one after another. Since it only enforces from the current latest migration, it’s compatible with apps that have merge migrations in their past.

django-linear-migrations also provides a tool for handling the merge conflicts it creates, in its rebase-migration command. Rather than repeat it here, I’ll point you to its documentation for an example.

If you too have struggled with merge migrations, please try out django-linear-migrations today!

Learn how to make your tests run quickly in my book Speed Up Your Django Tests.

One summary email a week, no spam, I pinky promise.

Related posts:

• Django and the N+1 Queries Problem
• How to Unit Test a Django Form
• Introducing time-machine, a New Python Library for Mocking the Current Time

Tags: django