Updated on February 21st, 2023 in #dev-environment

Django 4.1+ HTML Templates Are Cached by Default with DEBUG = True

Here's how you can fix HTML templates not being updated in development after upgrading to Django 4.1+.

Quick Jump: Demo Video

Want to skip to the code snippet solution that works for dev and prod? It’s down below.

You may have discovered that in development with DEBUG = True set, you’ve noticed that your HTML templates don’t get updated after making a change. That’s both your main project’s templates as well as your apps’ templates.

I tried a bunch of different things to pin point the problem:

• Using different code editors since Vim sometimes has issues with file watchers due to how it writes files to disk, I tried Nano and VSCode
• I use Docker, so I tried editing the files directly in the container instead of my dev box
• I configured my Django settings file to read DEBUG from an environment variable, so I tried hard coding it to True
• I use gunicorn in development and production, it has its own reload option which I hard coded to True instead of using an env var
• I confirmed both settings were correctly set by printing their values and seeing their output in the terminal as True
• Code was reloading properly if I edited Python files, CSS and JS also worked using Tailwind’s and esbuild’s watchers
• I disabled the esbuild and tailwind containers just to make sure their bind mounts weren’t overlapping and causing a ruckus with gunicorn’s watcher, it no had effect
• I have a similar example Docker project in Flask and it uses the same version of Python and uses it gunicorn too, everything worked there

The reloading behavior was also not very consistent. I found myself sometimes seeing updates if I edited the main project’s layout HTML file and also sometimes saw updates if I edited an app’s template such as the home page.

At that point I opened an issue in the project I was maintaining where this was first reported by someone in a different issue.

I left things off in the issue that I would investigate it further by checking out previous versions of the code where I know HTML template reloading worked but at the same time I also tweeted it out to see what the community thought.

In less than an hour, Jeff Triplett who is well known in the Django community pointed me to the Django 4.1+ release notes which mentions that HTML template caching is enabled by default even with DEBUG = True set. Here’s a snippet from the docs:

The cached template loader is now enabled in development, when DEBUG is True, and OPTIONS[‘loaders’] isn’t specified. You may specify OPTIONS[‘loaders’] to override this, if necessary.

Armed with that knowledge and after looking at the template loader options I landed on the idea that I can use a different set of loaders depending on what DEBUG is set to.

That code looks like this and it’s been commit to the project:

default_loaders = [
    "django.template.loaders.filesystem.Loader",
    "django.template.loaders.app_directories.Loader",
]

cached_loaders = [("django.template.loaders.cached.Loader", default_loaders)]

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": [os.path.join(BASE_DIR, "templates")],
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
            ],
            "loaders": default_loaders if DEBUG else cached_loaders,
        },
    },
]

Everything works in both development and production now based on what DEBUG is set to.

I also learned why I got inconsistent behavior. Before the above patch I sometimes saw HTML updates because if you edit a Python file (such as my settings config), that would trigger Django to invalidate the HTML template cache so you would see your updates.

That happened while I was occasionally editing my settings to convert the environment variables into hard coded values. It was driving me bonkers because sometimes I edited the file in Vim and other times in Nano and I kept getting different results.

The video below goes over the debugging process.

This adventure was a good reminder of 2 popular programming quotes. Caching is hard and given enough eyeballs, all bugs are shallow.

Demo Video

Timestamps

• 0:32 – Being notified of the issue and basic questions
• 1:58 – Demoing the problem
• 4:46 – Trying a different code editor in case Vim is causing file watcher issues
• 5:37 – Making sure the environment variables are being read correctly
• 8:02 – Ensuring the Django settings file is seeing the correct env value
• 10:17 – Updating a Python file will invalidate the HTML template cache
• 10:57 – I removed the asset containers, that didn’t help
• 11:48 – Creating an issue to document it and checking the TEMPLATES variable
• 13:58 – The next step was to do a bisection search to find the last known good state
• 15:45 – Throwing out a tweet and getting help in less than an hour
• 16:19 – Checking out the Django 4.1 release notes related to template caching
• 18:13 – Coming up with a solution that works in dev and prod
• 20:24 – Demoing the solution
• 21:05 – I use gunicorn in development and production

Did you end up getting hit by this after updating Django? Let me know below.