If you’ve ever thought about refactoring your Django app, then you might have found yourself needing to move a Django model around. There are several ways to move a Django model from one app to another using Django migrations, but unfortunately none of them are straightforward.
Moving models between Django apps is usually a very complicated task that involves copying data, changing constraints, and renaming objects. Because of these complications, the Django object-relational mapper (ORM) does not provide built-in migration operations that can detect and automate the entire process. Instead, the ORM provides a set of low-level migration operations that allow Django developers to implement the process themselves in the migration framework.
In this tutorial, you’ll learn:
- How to move a Django model from one app to another
- How to use advanced features of the Django migration command line interface (CLI), such as
sqlmigrate,showmigrations, andsqlsequencereset - How to produce and inspect a migration plan
- How to make a migration reversible and how to reverse migrations
- What introspection is and how Django uses it in migrations
After completing this tutorial, you’ll be able to choose the best approach for moving a Django model from one app to another based on your specific use case.
Free Bonus: Click here to get access to a free Django Learning Resources Guide (PDF) that shows you tips and tricks as well as common pitfalls to avoid when building Python + Django web applications.
Example Case: Move a Django Model to Another App
Throughout this tutorial, you’re going work on a store app. Your store is going to start with two Django apps:
catalog: This app is for storing data on products and product categories.sale: This app is for recording and tracking product sales.
After you finish setting up these two apps, you’re going to move a Django model called Product to a new app called product. In the process, you’ll face the following challenges:
- The model being moved has foreign key relationships with other models.
- Other models have foreign key relationships with the model being moved.
- The model being moved has an index on one of the fields (besides the primary key).
These challenges are inspired by real-life refactoring processes. After you overcome them, you’ll be ready to plan a similar migration process for your specific use case.
Setup: Prepare Your Environment
Before you start moving things around, you need to set up the initial state of your project. This tutorial uses Django 3 running on Python 3.8, but you can use similar techniques in other versions.
Set Up a Python Virtual Environment
First, create your virtual environment in a new directory:
$ mkdir django-move-model-experiment
$ cd django-move-model-experiment
$ python -m venv venv
For step-by-step instructions on creating a virtual environment, check out Python Virtual Environments: A Primer.
Create a Django Project
In your terminal, activate the virtual environment and install Django:
$ source venv/bin/activate
$ pip install django
Collecting django
Collecting pytz (from django)
Collecting asgiref~=3.2 (from django)
Collecting sqlparse>=0.2.2 (from django)
Installing collected packages: pytz, asgiref, sqlparse, django
Successfully installed asgiref-3.2.3 django-3.0.4 pytz-2019.3 sqlparse-0.3.1
You’re now ready to create your Django project. Use django-admin startproject to create a project called django-move-model-experiment:
$ django-admin startproject django-move-model-experiment
$ cd django-move-model-experiment
After running this command, you’ll see that Django created new files and directories. For more about how to start a new Django project, check out Starting a Django Project.
Create Django Apps
Now that you have a fresh Django project, create an app with your store’s product catalog:
$ python manage.py startapp catalog
Next, add the following models to the new catalog app:
# catalog/models.py
from django.db import models
class Category(models.Model):
name = models.CharField(max_length=100)
class Product(models.Model):
name = models.CharField(max_length=100, db_index=True)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
You’ve successfully created Category and Product models in your catalog app. Now that you have a catalog, you want to start selling your products. Create another app for sales:
$ python manage.py startapp sale
Add the following Sale model to the new sale app:
# sale/models.py
from django.db import models
from catalog.models import Product
class Sale(models.Model):
created = models.DateTimeField()
product = models.ForeignKey(Product, on_delete=models.PROTECT)
Notice that the Sale model references the Product model using a ForeignKey.
Generate and Apply Initial Migrations
To complete the setup, generate migrations and apply them:
$ python manage.py makemigrations catalog sale
Migrations for 'catalog':
catalog/migrations/0001_initial.py
- Create model Category
- Create model Product
Migrations for 'sale':
sale/migrations/0001_initial.py
- Create model Sale
$ python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, catalog, contenttypes, sale, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying catalog.0001_initial... OK
Applying sale.0001_initial... OK
Applying sessions.0001_initial... OK
For more about Django migrations check out Django Migrations: A Primer. With your migrations in place, you’re now ready to create some sample data!
Generate Sample Data
To make the migration scenario as realistic as possible, activate the Django shell from your terminal window:
$ python manage.py shell
Next, create the following objects:
>>> from catalog.models import Category, Product
>>> clothes = Category.objects.create(name='Clothes')
>>> shoes = Category.objects.create(name='Shoes')
>>> Product.objects.create(name='Pants', category=clothes)
>>> Product.objects.create(name='Shirt', category=clothes)
>>> Product.objects.create(name='Boots', category=shoes)
You created two categories, 'Shoes' and 'Clothes'. Next, you added two products, 'Pants' and 'Shirt', to the 'Clothes' category and one product, 'Boots', to the 'Shoes' category.
Congratulations! You’ve completed the setup for the initial state of your project. In a real-life scenario, this is where you would start planning your refactoring. Each of the three approaches presented in this tutorial will start from this point.
The Long Way: Copy the Data to a New Django Model
To kick things off, you’re going to take the long road:
- Create a new model
- Copy the data to it
- Drop the old table
This approach has some pitfalls that you should be aware of. You’ll explore them in detail in the following sections.
Create the New Model
Start by creating a new product app. From your terminal, execute the following command:
$ python manage.py startapp product
After running this command, you’ll notice a new directory called product was added to the project.
To register the new app with your existing Django project, add it to the list of INSTALLED_APPS in Django’s settings.py:
--- a/store/store/settings.py
+++ b/store/store/settings.py
@@ -40,6 +40,7 @@ INSTALLED_APPS = [
'catalog',
'sale',
+ 'product',
]
MIDDLEWARE = [
Your new product app is now registered with Django. Next, create a Product model in the new product app. You can copy the code from the catalog app:
# product/models.py
from django.db import models
from catalog.models import Category
class Product(models.Model):
name = models.CharField(max_length=100, db_index=True)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
Now that you’ve defined the model, try to generate migrations for it:
$ python manage.py makemigrations product
SystemCheckError: System check identified some issues:
ERRORS:
catalog.Product.category: (fields.E304) Reverse accessor for 'Product.category' clashes with reverse accessor for 'Product.category'.
HINT: Add or change a related_name argument to the definition for 'Product.category' or 'Product.category'.
product.Product.category: (fields.E304) Reverse accessor for 'Product.category' clashes with reverse accessor for 'Product.category'.
HINT: Add or change a related_name argument to the definition for 'Product.category' or 'Product.category'.
The error says that Django found two models with the same reverse accessor for the field category. This is because there are two models named Product that reference the Category model, creating a conflict.
When you add foreign keys to your model, Django creates a reverse accessor in the related model. In this case, the reverse accessor is products. The reverse accessor lets you access related objects like this: category.products.
The new model is the one you want to keep, so to resolve this conflict, remove the reverse accessor from the old model in catalog/models.py:
--- a/store/catalog/models.py
+++ b/store/catalog/models.py
@@ -7,4 +7,4 @@ class Category(models.Model):
class Product(models.Model):
name = models.CharField(max_length=100, db_index=True)
- category = models.ForeignKey(Category, on_delete=models.CASCADE)
+ category = models.ForeignKey(Category, on_delete=models.CASCADE, related_name='+')
The attribute related_name can be used to explicitly set a related name for a reverse accessor. Here, you use the special value +, which instructs Django not to create a reverse accessor.
Now generate a migration for the catalog app:
$ python manage.py makemigrations catalog
Migrations for 'catalog':
catalog/migrations/0002_auto_20200124_1250.py
- Alter field category on product
Don’t apply this migration yet! Once this change takes place, code that used the reverse accessor might break.
Now that there’s no conflict between the reverse accessors, try to generate the migrations for the new product app:
$ python manage.py makemigrations product
Migrations for 'product':
product/migrations/0001_initial.py
- Create model Product
Great! You’re ready to move on to the next step.
Copy the Data to the New Model
In the previous step, you created a new product app with a Product model that is identical to the model you want to move. The next step is to move the data from the old model to the new model.
To create a data migration, execute the following command from your terminal:
$ python manage.py makemigrations product --empty
Migrations for 'product':
product/migrations/0002_auto_20200124_1300.py
Edit the new migration file, and add the operation to copy the data from the old table:
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('product', '0001_initial'),
]
operations = [
migrations.RunSQL("""
INSERT INTO product_product (
id,
name,
category_id
)
SELECT
id,
name,
category_id
FROM
catalog_product;
""", reverse_sql="""
INSERT INTO catalog_product (
id,
name,
category_id
)
SELECT
id,
name,
category_id
FROM
product_product;
""")
]
To execute SQL in a migration, you use the special RunSQL migration command. The first argument is the SQL to apply. You also provide an action to reverse the migration using the reverse_sql argument.
Reversing a migration can come in handy when you discover a mistake and you want to roll back the change. Most built-in migration actions can be reversed. For example, the reverse action for adding a field is removing the field. The reverse action for creating a new table is dropping the table. It’s usually best to provide reverse_SQL to RunSQL so you can backtrack if something goes awry.
In this case, the forward migration operation inserts data from product_product into catalog_product. The backward operation will do the exact opposite, inserting data from catalog_product into product_product. By providing Django with the reverse operation, you’ll be able to reverse the migration in case of disaster.
At this point, you’re still halfway through the migration process. But there’s a lesson to be learned here, so go ahead and apply the migrations:
$ python manage.py migrate product
Operations to perform:
Apply all migrations: product
Running migrations:
Applying product.0001_initial... OK
Applying product.0002_auto_20200124_1300... OK
Before you move on to the next step, try to create a new product:
>>> from product.models import Product
>>> Product.objects.create(name='Fancy Boots', category_id=2)
Traceback (most recent call last):
File "/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "product_product_pkey"
DETAIL: Key (id)=(1) already exists.
When you use an auto-incrementing primary key, Django creates a sequence in the database to assign unique identifiers to new objects. Notice, for example, that you didn’t provide an ID for the new product. You normally wouldn’t want to provide an ID because you want the database to assign primary keys for you using a sequence. However, in this case, the new table gave the new product the ID 1 even though this ID already existed in the table.
So, what went wrong? When you copied the data to the new table, you didn’t sync the sequence. To sync the sequence, you can use another Django admin command called sqlsequencereset. The command produces a script to set the current value of the sequence based on existing data in the table. This command is often used for populating new models with preexisting data.
Use sqlsequencereset to produce a script to sync the sequence:
$ python manage.py sqlsequencereset product
BEGIN;
SELECT setval(pg_get_serial_sequence('"product_product"','id'), coalesce(max("id"), 1), max("id") IS NOT null)
FROM "product_product";
COMMIT;
The script generated by the command is database specific. In this case, the database is PostgreSQL. The script sets the current value of the sequence to the next value that the sequence should yield, which is the maximum ID in the table plus one.
To finish up, add the snippet to the data migration:
--- a/store/product/migrations/0002_auto_20200124_1300.py
+++ b/store/product/migrations/0002_auto_20200124_1300.py
@@ -22,6 +22,8 @@ class Migration(migrations.Migration):
category_id
FROM
catalog_product;
+
+ SELECT setval(pg_get_serial_sequence('"product_product"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "product_product";
""", reverse_sql="""
INSERT INTO catalog_product (
id,
The snippet will sync the sequence when you apply the migration, solving the sequence issue you ran into above.
This detour to learn about syncing sequences has created a little mess in your code. To clean it up, delete the data in the new model from the Django shell:
>>> from product.models import Product
>>> Product.objects.all().delete()
(3, {'product.Product': 3})
Now that the data you copied is deleted, you can reverse the migration. To reverse a migration, you migrate to a previous migration:
$ python manage.py showmigrations product
product
[X] 0001_initial
[X] 0002_auto_20200124_1300
$ python manage.py migrate product 0001_initial
Operations to perform:
Target specific migration: 0001_initial, from product
Running migrations:
Rendering model states... DONE
Unapplying product.0002_auto_20200124_1300... OK
You first used the command showmigrations to list the migrations applied to the app product. The output shows that both migrations were applied. You then reversed migration 0002_auto_20200124_1300 by migrating to the prior migration, 0001_initial.
If you execute showmigrations again, then you’ll see that the second migration is no longer marked as applied:
$ python manage.py showmigrations product
product
[X] 0001_initial
[ ] 0002_auto_20200124_1300
The empty box confirms that the second migration has been reversed. Now that you have a clean slate, run the migrations with the new code:
$ python manage.py migrate product
Operations to perform:
Apply all migrations: product
Running migrations:
Applying product.0002_auto_20200124_1300... OK
The migration was successfully applied. Make sure you can now create a new Product in the Django shell:
>>> from product.models import Product
>>> Product.objects.create(name='Fancy Boots', category_id=2)
<Product: Product object (4)>
Amazing! Your hard work paid off, and you’re ready for the next step.
Update Foreign Keys to the New Model
The old table currently has other tables referencing it using a ForeignKey field. Before you can remove the old model, you need to change the models referencing the old model so that they reference the new model.
One model that still references the old model is Sale in the sale app. Change the foreign key in the Sale model to reference the new Product model:
--- a/store/sale/models.py
+++ b/store/sale/models.py
@@ -1,6 +1,6 @@
from django.db import models
-from catalog.models import Product
+from product.models import Product
class Sale(models.Model):
created = models.DateTimeField()
Generate the migration and apply it:
$ python manage.py makemigrations sale
Migrations for 'sale':
sale/migrations/0002_auto_20200124_1343.py
- Alter field product on sale
$ python manage.py migrate sale
Operations to perform:
Apply all migrations: sale
Running migrations:
Applying sale.0002_auto_20200124_1343... OK
The Sale model now references the new Product model in the product app. Because you already copied all the data to the new model, there are no constraint violations.
Delete the Old Model
The previous step eliminated all references to the old Product model. It’s now safe to remove the old model from the catalog app:
--- a/store/catalog/models.py
+++ b/store/catalog/models.py
@@ -3,8 +3,3 @@ from django.db import models
class Category(models.Model):
name = models.CharField(max_length=100)
-
-
-class Product(models.Model):
- name = models.CharField(max_length=100, db_index=True)
- category = models.ForeignKey(Category, on_delete=models.CASCADE, related_name='+')
Generate a migration but don’t apply it yet:
$ python manage.py makemigrations
Migrations for 'catalog':
catalog/migrations/0003_delete_product.py
- Delete model Product
To make sure the old model is deleted only after the data is copied, add the following dependency:
--- a/store/catalog/migrations/0003_delete_product.py
+++ b/store/catalog/migrations/0003_delete_product.py
@@ -7,6 +7,7 @@ class Migration(migrations.Migration):
dependencies = [
('catalog', '0002_auto_20200124_1250'),
+ ('sale', '0002_auto_20200124_1343'),
]
operations = [
Adding this dependency is extremely important. Skipping this step can have terrible consequences, including lost data. For more about migration files and dependencies between migrations, check out Digging Deeper Into Django Migrations.
Note: The name of the migration includes the date and the time it was generated. If you’re following along with your own code, then these parts of the name will be different.
Now that you’ve added the dependency, apply the migration:
$ python manage.py migrate catalog
Operations to perform:
Apply all migrations: catalog
Running migrations:
Applying catalog.0003_delete_product... OK
The transfer is now complete! You’ve successfully moved the Product model from the catalog app to the new product app by creating a new model and coping the data to it.
Bonus: Reverse the Migrations
One of the benefits of Django migrations is that they’re reversible. What does it mean for a migration to be reversible? If you make a mistake, then you can reverse the migration and the database will revert back to the state from before the migration was applied.
Remember how you previously provided reverse_sql to RunSQL? Well, this is where it pays off.
Apply all the migrations on a fresh database:
$ python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, catalog, contenttypes, product, sale, sessions
Running migrations:
Applying product.0001_initial... OK
Applying product.0002_auto_20200124_1300... OK
Applying sale.0002_auto_20200124_1343... OK
Applying catalog.0003_delete_product... OK
Now, reverse them all using the special keyword zero:
$ python manage.py migrate product zero
Operations to perform:
Unapply all migrations: product
Running migrations:
Rendering model states... DONE
Unapplying catalog.0003_delete_product... OK
Unapplying sale.0002_auto_20200124_1343... OK
Unapplying product.0002_auto_20200124_1300... OK
Unapplying product.0001_initial... OK
The database is now back to its original state. If you deploy this version and discover a mistake, then you can reverse it!
Handle Special Cases
When you’re moving models from one app to another, some Django features may require special attention. In particular, adding or modifying database constraints and using generic relations both require extra care.
Modifying Constraints
Adding constraints to tables with data can be a dangerous operation to perform on a live system. To add a constraint, the database must first verify it. During the verification, the database acquires a lock on the table, which might prevent other operations until the process completes.
Some constraints, such as NOT NULL and CHECK, may require a full scan of the table to verify that the new data is valid. Other constraints, such as FOREIGN KEY, require validation with another table, which can take some time depending on the size of the referenced table.
Handling Generic Relations
If you’re using generic relations, then you might need an extra step. Generic relations use both the primary key and the content type ID of the model to reference a row in any model table. The old model and the new model do not have the same content type ID, so generic connections can break. This can sometimes go unnoticed because the integrity of generic foreign keys is not enforced by the database.
There are two ways to deal with generic foreign keys:
- Update the content type ID of the new model to that of the old model.
- Update the content type ID of any referencing tables to that of the new model.
Either way you choose, make sure to test it properly before deploying to production.
Summary: Pros and Cons of Copying the Data
Moving a Django model to another app by copying the data has its advantages and disadvantages. Here are some of the pros associated with this approach:
- It’s supported by the ORM: Performing this transition using built-in migration operations guarantees proper database support.
- It’s reversible: Reversing this migration is possible if necessary.
Here are some of the cons associated with this approach:
- It’s slow: Copying large amounts of data can take time.
- It requires downtime: Changing the data in the old table while it’s being copied to the new table will cause data loss during the transition. To prevent this from happening, downtime is necessary.
- It requires manual work to sync the database: Loading data to existing tables requires syncing sequences and generic foreign keys.
As you’ll see in the following sections, using this approach to move a Django model to another app takes much longer than other approaches.
The Short Way: Reference the New Django Model to the Old Table
In the previous approach, you copied all the data to the new table. The migration required downtime, and it could take a long time to complete depending on how much data there is to copy.
What if, instead of copying the data, you changed the new model to reference the old table?
Create the New Model
This time, you’re going to make all the changes to the models at once, and then let Django generate all the migrations.
First, remove the Product model from the catalog app:
--- a/store/catalog/models.py
+++ b/store/catalog/models.py
@@ -3,8 +3,3 @@ from django.db import models
class Category(models.Model):
name = models.CharField(max_length=100)
-
-
-class Product(models.Model):
- name = models.CharField(max_length=100, db_index=True)
- category = models.ForeignKey(Category, on_delete=models.CASCADE)
You’ve removed the Product model from the catalog app. Now move the Product model to the new product app:
# store/product/models.py
from django.db import models
from catalog.models import Category
class Product(models.Model):
name = models.CharField(max_length=100, db_index=True)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
Now that the Product model exists in the product app, you can change any references to the old Product model to reference the new Product model. In this case, you’ll need to change the foreign key in sale to reference product.Product:
--- a/store/sale/models.py
+++ b/store/sale/models.py
@@ -1,6 +1,6 @@
from django.db import models
-from catalog.models import Product
+from product.models import Product
class Sale(models.Model):
created = models.DateTimeField()
Before moving on to generating the migrations, you need to make one more small change to the new Product model:
--- a/store/product/models.py
+++ b/store/product/models.py
@@ -5,3 +5,6 @@ from catalog.models import Category
class Product(models.Model):
name = models.CharField(max_length=100, db_index=True)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
+
+ class Meta:
+ db_table = 'catalog_product'
Django models have a Meta option called db_table. With this option, you can provide a table name to use in place of the one generated by Django. This option is most commonly used when setting up the ORM on an existing database schema in which the table names don’t match Django’s naming convention.
In this case, you set the name of the table in the product app to reference the existing table in the catalog app.
To complete the setup, generate migrations:
$ python manage.py makemigrations sale product catalog
Migrations for 'catalog':
catalog/migrations/0002_remove_product_category.py
- Remove field category from product
catalog/migrations/0003_delete_product.py
- Delete model Product
Migrations for 'product':
product/migrations/0001_initial.py
- Create model Product
Migrations for 'sale':
sale/migrations/0002_auto_20200104_0724.py
- Alter field product on sale
Before you move forward, produce a migration plan using the --plan flag:
$ python manage.py migrate --plan
Planned operations:
catalog.0002_remove_product_category
Remove field category from product
product.0001_initial
Create model Product
sale.0002_auto_20200104_0724
Alter field product on sale
catalog.0003_delete_product
Delete model Product
The output of the command lists the order in which Django is going to apply the migrations.
Eliminate Changes to the Database
The main benefit of this approach is that you don’t actually make any changes to the database, only to the code. To eliminate changes to the database, you can use the special migration operation SeparateDatabaseAndState.
SeparateDatabaseAndState can be used to modify the actions Django performs during a migration. For more about how to use SeparateDatabaseAndState, check out How to Create an Index in Django Without Downtime.
If you look at the contents of the migrations Django generated, then you’ll see that Django creates a new model and deletes the old one. If you execute these migrations, then the data will be lost, and the table will be created empty. To avoid this, you need to make sure Django does not make any changes to the database during the migration.
You can eliminate changes to the database by wrapping each migration operation in a SeparateDatabaseAndState operation. To tell Django not to apply any changes to the database, you can set db_operations to an empty list.
You plan on reusing the old table, so you need to prevent Django from dropping it. Before dropping a model, Django will drop the fields referencing the model. So, first, prevent Django from dropping the foreign key from sale to product:
--- a/store/catalog/migrations/0002_remove_product_category.py
+++ b/store/catalog/migrations/0002_remove_product_category.py
@@ -10,8 +10,14 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.RemoveField(
- model_name='product',
- name='category',
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.RemoveField(
+ model_name='product',
+ name='category',
+ ),
+ ],
+ # You're reusing the table, so don't drop it
+ database_operations=[],
),
]
Now that Django has handled the related objects, it can drop the model. You want to keep the Product table, so prevent Django from dropping it:
--- a/store/catalog/migrations/0003_delete_product.py
+++ b/store/catalog/migrations/0003_delete_product.py
@@ -11,7 +11,13 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.DeleteModel(
- name='Product',
- ),
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.DeleteModel(
+ name='Product',
+ ),
+ ],
+ # You want to reuse the table, so don't drop it
+ database_operations=[],
+ )
]
You used database_operations=[] to prevent Django from dropping the table. Next, prevent Django from creating the new table:
--- a/store/product/migrations/0001_initial.py
+++ b/store/product/migrations/0001_initial.py
@@ -13,15 +13,21 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.CreateModel(
- name='Product',
- fields=[
- ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
- ('name', models.CharField(db_index=True, max_length=100)),
- ('category', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='catalog.Category')),
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.CreateModel(
+ name='Product',
+ fields=[
+ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+ ('name', models.CharField(db_index=True, max_length=100)),
+ ('category', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='catalog.Category')),
+ ],
+ options={
+ 'db_table': 'catalog_product',
+ },
+ ),
],
- options={
- 'db_table': 'catalog_product',
- },
- ),
+ # You reference an existing table
+ database_operations=[],
+ )
]
Here, you used database_operations=[] to prevent Django from creating a new table. Finally, you want to prevent Django from re-creating the foreign key constraint from Sale to the new Product model. Since you’re reusing the old table, the constraint remains in place:
--- a/store/sale/migrations/0002_auto_20200104_0724.py
+++ b/store/sale/migrations/0002_auto_20200104_0724.py
@@ -12,9 +12,14 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.AlterField(
- model_name='sale',
- name='product',
- field=models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, to='product.Product'),
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.AlterField(
+ model_name='sale',
+ name='product',
+ field=models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, to='product.Product'),
+ ),
+ ],
+ database_operations=[],
),
]
Now that you’re done editing the migration files, apply the migrations:
$ python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, catalog, contenttypes, product, sale, sessions
Running migrations:
Applying catalog.0002_remove_product_category... OK
Applying product.0001_initial... OK
Applying sale.0002_auto_20200104_0724... OK
Applying catalog.0003_delete_product... OK
At this point, your new model points to the old table. Django didn’t make any changes to the database, and all the changes were made to Django’s model state in the code. But before you call this a success and move forward, it’s worth confirming that the new model’s state matches that of the database.
Bonus: Make Changes to the New Model
To make sure the state of the models is consistent with the state of the database, try to make a change to the new model, and make sure Django detects it correctly.
The Product model defines an index on the name field. Remove that index:
--- a/store/product/models.py
+++ b/store/product/models.py
@@ -3,7 +3,7 @@ from django.db import models
from catalog.models import Category
class Product(models.Model):
- name = models.CharField(max_length=100, db_index=True)
+ name = models.CharField(max_length=100)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
class Meta:
You removed the index by eliminating db_index=True. Next, generate a migration:
$ python manage.py makemigrations
Migrations for 'product':
product/migrations/0002_auto_20200104_0856.py
- Alter field name on product
Before moving forward, inspect the SQL generated by Django for this migration:
$ python manage.py sqlmigrate product 0002
BEGIN;
--
-- Alter field name on product
--
DROP INDEX IF EXISTS "catalog_product_name_924af5bc";
DROP INDEX IF EXISTS "catalog_product_name_924af5bc_like";
COMMIT;
Great! Django detected the old index, as shown by the "catalog_*" prefix. Now you can execute the migration:
$ python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, catalog, contenttypes, product, sale, sessions
Running migrations:
Applying product.0002_auto_20200104_0856... OK
Make sure you got the expected result in the database:
django_migration_test=# \d catalog_product
Table "public.catalog_product"
Column | Type | Nullable | Default
-------------+------------------------+----------+---------------------------------------------
id | integer | not null | nextval('catalog_product_id_seq'::regclass)
name | character varying(100) | not null |
category_id | integer | not null |
Indexes:
"catalog_product_pkey" PRIMARY KEY, btree (id)
"catalog_product_category_id_35bf920b" btree (category_id)
Foreign-key constraints:
"catalog_product_category_id_35bf920b_fk_catalog_category_id"
FOREIGN KEY (category_id) REFERENCES catalog_category(id)
DEFERRABLE INITIALLY DEFERRED
Referenced by:
TABLE "sale_sale" CONSTRAINT "sale_sale_product_id_18508f6f_fk_catalog_product_id"
FOREIGN KEY (product_id) REFERENCES catalog_product(id)
DEFERRABLE INITIALLY DEFERRED
Success! The index on the name column was dropped.
Summary: Pros and Cons of Changing the Model Reference
Changing the model to reference another model has its advantages and disadvantages. Here are some of the pros associated with this approach:
- It’s fast: This approach doesn’t make any changes to the database, so it’s very fast.
- It doesn’t require downtime: This approach doesn’t require copying data, so it can be performed on a live system without downtime.
- It’s reversible: It’s possible to reverse this migration if necessary.
- It’s supported by the ORM: Performing this transition using built-in migration operations guarantees proper database support.
- It doesn’t require any sync to the database: With this approach, related objects, such as indices and sequences, remain unchanged.
The only major con to this approach is that it breaks the naming convention. Using the existing table means the table will still use the name of the old app.
Notice how this approach is much more straightforward than copying the data.
The Django Way: Rename the Table
In the previous example, you made the new model reference the old table in the database. As a result, you broke the naming convention used by Django. In this approach, you do the reverse: You make the old table reference the new model.
More specifically, you create the new model and generate a migration for it. You then take the name of the new table from the migration Django created and, instead of creating the table for the new model, you rename the old table to the name of the new table using the special migration operation AlterModelTable.
Create the New Model
Just like before, you start by creating a new product app to make all the changes at once. First, remove the Product model from the catalog app:
--- a/store/catalog/models.py
+++ b/store/catalog/models.py
@@ -3,8 +3,3 @@ from django.db import models
class Category(models.Model):
name = models.CharField(max_length=100)
-
-
-class Product(models.Model):
- name = models.CharField(max_length=100, db_index=True)
- category = models.ForeignKey(Category, on_delete=models.CASCADE)
You’ve eliminated Product from catalog. Next, move the Product model to a new product app:
# store/product/models.py
from django.db import models
from catalog.models import Category
class Product(models.Model):
name = models.CharField(max_length=100, db_index=True)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
The Product model now exists in your product app. Now change the foreign key in Sale to reference product.Product:
--- a/store/sale/models.py
+++ b/store/sale/models.py
@@ -1,6 +1,6 @@
from django.db import models
-from catalog.models import Product
+from product.models import Product
class Sale(models.Model):
created = models.DateTimeField()
--- a/store/store/settings.py
+++ b/store/store/settings.py
@@ -40,6 +40,7 @@ INSTALLED_APPS = [
'catalog',
'sale',
+ 'product',
]
Next, let Django generate migrations for you:
$ python manage.py makemigrations sale catalog product
Migrations for 'catalog':
catalog/migrations/0002_remove_product_category.py
- Remove field category from product
catalog/migrations/0003_delete_product.py
- Delete model Product
Migrations for 'product':
product/migrations/0001_initial.py
- Create model Product
Migrations for 'sale':
sale/migrations/0002_auto_20200110_1304.py
- Alter field product on sale
You want to prevent Django from dropping the table because you intend to rename it.
To get the name of the Product model in the product app, generate the SQL for the migration that creates Product:
$ python manage.py sqlmigrate product 0001
BEGIN;
--
-- Create model Product
--
CREATE TABLE "product_product" ("id" serial NOT NULL PRIMARY KEY, "name" varchar(100) NOT NULL, "category_id" integer NOT NULL);
ALTER TABLE "product_product" ADD CONSTRAINT "product_product_category_id_0c725779_fk_catalog_category_id" FOREIGN KEY ("category_id") REFERENCES "catalog_category" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "product_product_name_04ac86ce" ON "product_product" ("name");
CREATE INDEX "product_product_name_04ac86ce_like" ON "product_product" ("name" varchar_pattern_ops);
CREATE INDEX "product_product_category_id_0c725779" ON "product_product" ("category_id");
COMMIT;
The name of the table that Django generated for the Product model in the product app is product_product.
Rename the Old Table
Now that you have the name Django generated for the model, you’re ready to rename the old table. To drop the Product model from the catalog app, Django created two migrations:
catalog/migrations/0002_remove_product_categoryremoves the foreign key from the table.catalog/migrations/0003_delete_productdrops the model.
Before you can rename the table, you want to prevent Django from dropping the foreign key to Category:
--- a/store/catalog/migrations/0002_remove_product_category.py
+++ b/store/catalog/migrations/0002_remove_product_category.py
@@ -10,8 +10,13 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.RemoveField(
- model_name='product',
- name='category',
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.RemoveField(
+ model_name='product',
+ name='category',
+ ),
+ ],
+ database_operations=[],
),
]
Using SeparateDatabaseAndState with database_operations set to an empty list prevents Django from dropping the column.
Django provides a special migration operation, AlterModelTable, to rename a table for a model. Edit the migration that drops the old table, and instead rename the table to product_product:
--- a/store/catalog/migrations/0003_delete_product.py
+++ b/store/catalog/migrations/0003_delete_product.py
@@ -11,7 +11,17 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.DeleteModel(
- name='Product',
- ),
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.DeleteModel(
+ name='Product',
+ ),
+ ],
+ database_operations=[
+ migrations.AlterModelTable(
+ name='Product',
+ table='product_product',
+ ),
+ ],
+ )
]
You used SeparateDatabaseAndState along with AlterModelTable to provide Django with a different migration action to execute in the database.
Next, you need to prevent Django from creating a table for the new Product model. Instead, you want it to use the table you renamed. Make the following change to the initial migration in the product app:
--- a/store/product/migrations/0001_initial.py
+++ b/store/product/migrations/0001_initial.py
@@ -13,12 +13,18 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.CreateModel(
- name='Product',
- fields=[
- ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
- ('name', models.CharField(db_index=True, max_length=100)),
- ('category', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='catalog.Category')),
- ],
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.CreateModel(
+ name='Product',
+ fields=[
+ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+ ('name', models.CharField(db_index=True, max_length=100)),
+ ('category', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='catalog.Category')),
+ ],
+ ),
+ ],
+ # Table already exists. See catalog/migrations/0003_delete_product.py
+ database_operations=[],
),
]
The migration creates the model in Django’s state, but it doesn’t create the table in the database because of the line database_operations=[]. Remember when you renamed the old table to product_product? By renaming the old table to the name Django would have generated for the new model, you force Django to use the old table.
Finally, you want to prevent Django from re-creating the foreign key constraint in the Sale model:
--- a/store/sale/migrations/0002_auto_20200110_1304.py
+++ b/store/sale/migrations/0002_auto_20200110_1304.py
@@ -12,9 +12,15 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.AlterField(
- model_name='sale',
- name='product',
- field=models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, to='product.Product'),
- ),
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.AlterField(
+ model_name='sale',
+ name='product',
+ field=models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, to='product.Product'),
+ ),
+ ],
+ # You're reusing an existing table, so do nothing
+ database_operations=[],
+ )
]
You’re now ready to run the migration:
$ python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, catalog, contenttypes, product, sale, sessions
Running migrations:
Applying catalog.0002_remove_product_category... OK
Applying product.0001_initial... OK
Applying sale.0002_auto_20200110_1304... OK
Applying catalog.0003_delete_product... OK
Great! The migration was successful. But before you move on, make sure it can be reversed:
$ python manage.py migrate catalog 0001
Operations to perform:
Target specific migration: 0001_initial, from catalog
Running migrations:
Rendering model states... DONE
Unapplying catalog.0003_delete_product... OK
Unapplying sale.0002_auto_20200110_1304... OK
Unapplying product.0001_initial... OK
Unapplying catalog.0002_remove_product_category... OK
Amazing! The migration is completely reversible.
Note: AlterModelTable is generally preferable to RunSQL for a few reasons.
First, AlterModelTable can handle many-to-many relations between fields whose names are based on the model’s name. Using RunSQL to rename the tables may require some additional work.
Also, built-in migration operations such as AlterModelTable are database agnostic, whereas RunSQL is not. If, for example, your app needs to work on multiple database engines, then you might run into some trouble writing SQL that’s compatible with all database engines.
Bonus: Understand Introspection
The Django ORM is an abstraction layer that translates Python types to database tables and vice versa. For example, when you created the model Product in the product app, Django created a table called product_product. Besides tables, the ORM creates other database objects, such as indices, constraints, sequences, and more. Django has a naming convention for all of these objects based on the names of the app and model.
To get a better sense of what it looks like, inspect the table catalog_category in the database:
django_migration_test=# \d catalog_category
Table "public.catalog_category"
Column | Type | Nullable | Default
--------+------------------------+----------+----------------------------------------------
id | integer | not null | nextval('catalog_category_id_seq'::regclass)
name | character varying(100) | not null |
Indexes:
"catalog_category_pkey" PRIMARY KEY, btree (id)
The table was generated by Django for the Category model in the app catalog, hence the name catalog_category. You can also notice a similar naming convention for the other database objects.
catalog_category_pkeyrefers to a primary key index.catalog_category_id_seqrefers to a sequence to generate values for the primary key fieldid.
Next, inspect the table of the Product model you moved from catalog to product:
django_migration_test=# \d product_product
Table "public.product_product"
Column | Type | Nullable | Default
-------------+------------------------+----------+---------------------------------------------
id | integer | not null | nextval('catalog_product_id_seq'::regclass)
name | character varying(100) | not null |
category_id | integer | not null |
Indexes:
"catalog_product_pkey" PRIMARY KEY, btree (id)
"catalog_product_category_id_35bf920b" btree (category_id)
"catalog_product_name_924af5bc" btree (name)
"catalog_product_name_924af5bc_like" btree (name varchar_pattern_ops)
Foreign-key constraints:
"catalog_product_category_id_35bf920b_fk_catalog_category_id"
FOREIGN KEY (category_id)
REFERENCES catalog_category(id)
DEFERRABLE INITIALLY DEFERRED
At first glance, there are more related objects. A closer look, however, reveals that the names of the related objects are not consistent with the name of the table. For example, the name of the table is product_product, but the name of the primary key constraint is catalog_product_pkey. You copied the model from the app called catalog, so that must mean that the migration operation AlterModelTable does not change the names of all related database objects.
To better understand how AlterModelTable works, inspect the SQL generated by this migration operation:
$ python manage.py sqlmigrate catalog 0003
BEGIN;
--
-- Custom state/database change combination
--
ALTER TABLE "catalog_product" RENAME TO "product_product";
COMMIT;
What this shows is that AlterModelTable only renames tables. If that’s the case, then what would happen if you tried to make a change to one of the database objects related to the tables of these objects? Would Django be able to handle these changes?
To find out, try to remove the index on the field name in the Product model:
--- a/store/product/models.py
+++ b/store/product/models.py
@@ -3,5 +3,5 @@ from django.db import models
from catalog.models import Category
class Product(models.Model):
- name = models.CharField(max_length=100, db_index=True)
+ name = models.CharField(max_length=100, db_index=False)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
Next, generate a migration:
$ python manage.py makemigrations
Migrations for 'product':
product/migrations/0002_auto_20200110_1426.py
- Alter field name on product
The command succeeded—that’s a good sign. Now inspect the generated SQL:
$ python manage.py sqlmigrate product 0002
BEGIN;
--
-- Alter field name on product
--
DROP INDEX IF EXISTS "catalog_product_name_924af5bc";
DROP INDEX IF EXISTS "catalog_product_name_924af5bc_like";
COMMIT;
The generated SQL commands drop the index catalog_product_name_924af5bc. Django was able to detect the existing index even though it’s not consistent with the name of the table. This is known as introspection.
Introspection is used internally by the ORM, so you won’t find much documentation about it. Each database backend contains an introspection module that can identify database objects based on their properties. The introspection module would usually use the metadata tables provided by the database. Using introspection, the ORM can manipulate objects without relying on the naming convention. This is how Django was able to detect the name of the index to drop.
Summary: Pros and Cons of Renaming the Table
Renaming the table has its advantages and disadvantages. Here are some of the pros associated with this approach:
- It’s fast: This approach only renames database objects, so it’s very fast.
- It doesn’t require downtime: With this approach, database objects are locked only for a short time while they are renamed, so it can be performed on a live system without downtime.
- It’s reversible: Reversing this migration is possible if necessary.
- It’s supported by the ORM: Performing this transition using built-in migration operations guarantees proper database support.
The only potential con associated with this approach is that it breaks the naming convention. Renaming only the tables means that the names of other database objects will be inconsistent with Django’s naming convention. This may cause some confusion when working with the database directly. However, Django can still use introspection to identify and manage these objects, so this is not a major concern.
Guidelines: Choose the Best Approach
In this tutorial, you’ve learned how to move a Django model from one app to another in three different ways. Here’s a comparison of the approaches described in this tutorial:
| Metric | Copy Data | Change Table | Rename Table |
|---|---|---|---|
| Fast | ✗ | ✔️ | ✔️ |
| No downtime | ✗ | ✔️ | ✔️ |
| Sync related objects | ✗ | ✔️ | ✔️ |
| Preserve naming convention | ✔️ | ✗ | ✔️ |
| Built-in ORM support | ✔️ | ✔️ | ✔️ |
| Reversible | ✔️ | ✔️ | ✔️ |
Note: The table above suggests that renaming the table preserves Django’s naming convention. While this is not strictly true, you learned earlier that Django can use introspection to overcome the naming issues associated with this approach.
Each of the approaches above has its own advantages and disadvantages. So, which approach should you use?
As a general rule of thumb, you should copy the data when you’re working with a small table and you can afford some downtime. Otherwise, your best bet is to rename the table and reference the new model to it.
That said, every project has its own unique requirements. You should choose whichever approach makes the most sense for you and your team.
Conclusion
After reading this tutorial, you’re in a better position to make the right decision about how to move a Django model to another app based on your specific use case, restrictions, and requirements.
In this tutorial, you’ve learned:
- How to move a Django model from one app to another
- How to use advanced features of the Django migration CLI, such as
sqlmigrate,showmigrations, andsqlsequencereset - How to produce and inspect a migration plan
- How to make a migration reversible, and how to reverse migrations
- What introspection is and how Django uses it in migrations
To dive deeper, check out the full range of database tutorials and Django tutorials.
