Django: Avoid database queries in template context processors

Time for a little game of moving templating pieces around…

Django’s template engine allows you to augment template contexts with context processors. These are functions that take the current request and return a dictionary to be merged into the context:

from example.models import HotDog
from example.models import HotDogState


def hot_dog_stats(request):
    return {
        "hot_dogs_eaten": HotDog.objects.filter(
            state=HotDogState.EATEN,
        ).count(),
    }

You enable context processors by pointing to them in the TEMPLATES setting:

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "OPTIONS": {
            "context_processors": [
                ...,
                "example.context_processors.hot_dog_stats",
                ...,
            ],
        },
    },
]

Django’s startproject template includes a few context processors, such as the one from django.contrib.auth that adds the user and perms variables.

Context processors are convenient for adding extra “global variables” to templates. But because they run on every top-level template render, they need to be robust and performant.

In particular, avoid running database queries within them. Database queries within a context processor can have at least two negative consequences:

  1. Performance impact

    If a context processor fetches data that is not used when rendering a page, the resources spent fetching the data will have been wasted.

    Since most sites contain different types of pages, pretty much any query run in a context processor will be wasted on some pages. For example, your Django Admin pages won’t use the same variables as other parts of your site.

  2. Breakage of your custom error view

    Django allows you to customize your “Internal Server Error” error page with either:

    If you use only a custom error template, Django renders it without a request context, so context processors don’t apply (source). But if you create a handler500 view that renders a template in the usual way, your context processors will run, for example:

    from http import HTTPStatus

from django.shortcuts import render


def server_error(request):
    return render(
        request,
        "server-error.html",
        status=HTTPStatus.INTERNAL_SERVER_ERROR,
    )

    Django will call your handler500 view for all types of errors, including database errors. So your database connection may potentially be in an unusable state. In such cases, if any of your context processors run database queries, then your error template will fail to render, and Django will fall back the plain text error page:

    A server error occurred.  Please contact the administrator.

    Uh oh.

    That is not a good user experience. Indeed, it’s the kind of obscure, generic message you create a custom error view to avoid.

So you want to avoid queries within context processors. How can you do that? Below are three techniques to run your queries only when needed.

Technique #1: Use a custom context wrapper

Rather than implicitly add the variables with a context processor, you could add them explicitly at each call to render a template:

from django.shortcuts import render

from example.models import HotDog
from example.models import HotDogState


def base_context(request):
    return {
        "hot_dogs_eaten": HotDog.objects.filter(
            state=HotDogState.EATEN,
        ).count(),
    }


def hot_dog_dashboard(request):
    ...
    return render(
        request,
        "hot-dog-dashboard.html",
        {
            **base_context(request),
            "mustards": mustards,
            # ...
        },
    )

To save lines and make it easier to remember the base context, you could push adding the variables into a render() wrapper:

from django.shortcuts import render as base_render

from example.models import HotDog
from example.models import HotDogState


def render(request, template_name, context):
    context["hot_dogs_eaten"] = HotDog.objects.filter(
        state=HotDogState.EATEN,
    ).count()
    return base_render(request, template_name, context)


def hot_dog_dashboard(request):
    ...
    return render(
        request,
        "hot-dog-dashboard.html",
        {
            "mustards": mustards,
            # ...
        },
    )

Then all your views within that part of your site can use the wrapper.

If you use class-based views, you’ll instead want to create a custom TemplateView class:

from django.views.generic.base import TemplateView as BaseTemplateView

from example.models import HotDog
from example.models import HotDogState


class TemplateView(BaseTemplateView):
    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context["hot_dogs_eaten"] = HotDog.objects.filter(
            state=HotDogState.EATEN,
        ).count()
        return context


class HotDogDashboardView(TemplateView):
    template_name = "hot-dog-dashboard.html"

    ...

(This may need work with multiple inheritance if you use other base view classes.)

This approach is appreciably explicit. But it can still lead to wasted queries on some pages if you aren’t careful.

Technique #2: Lazy-fy the queries

When you use a variable in a Django template:

{{ hot_dogs_eaten }}

…if the referred-to-object is callable, then Django will call it and use the returned value. (Unless it has truthy alters_data or do_not_call_in_templates attributes.)

You can use this behaviour to make queries lazy by wrapping them up in a callable. The simplest option is to use a lambda:

from example.models import HotDog
from example.models import HotDogState


def hot_dog_stats(request):
    return {
        "hot_dogs_eaten": lambda: HotDog.objects.filter(
            state=HotDogState.EATEN,
        ).count(),
    }

The lambda does defer the query until used. But, if rendering uses the same variable twice, the lambda will be called twice and thus execute the query twice. To cache the result after the first call, you can use functools.cache():

from functools import cache

from example.models import HotDog
from example.models import HotDogState


def hot_dog_stats(request):
    return {
        "hot_dogs_eaten": cache(
            lambda: HotDog.objects.filter(
                state=HotDogState.EATEN,
            ).count()
        ),
    }

Although I find it clearer to use the decorator form of cache with a vanilla function:

from functools import cache

from example.models import HotDog
from example.models import HotDogState


def stats(request):
    @cache
    def hot_dogs_eaten():
        return HotDog.objects.filter(
            state=HotDogState.EATEN,
        ).count()

    return {
        "hot_dogs_eaten": hot_dogs_eaten,
    }

Normal functions have a few advantages over lambda, such as having a name that appears in stack traces.

You can also combine this lazy-fication technique with the previous “base context” technique.

The main downside to this techinque is that it’s a bit more advanced. Django’s callable variable behaviour is a less-used feature, so other developers may be unaware of why the caching exists.

Technique #3: Move to an inclusion tag

If the variables from your context processor only appear in a particular template fragment, then you can create an inclusion tag. Inclusion tags are custom tag functions that can render an included template with a new computed context.

For example, below is an inclusion tag that renders hot dog statistics. This would need to live in a template tags file like example/templatetags/hot_dog_tags.py.

from django import template

from example.models import HotDog
from example.models import HotDogState

register = template.Library()


@register.inclusion_tag("includes/hot-dog-stats.html")
def hot_dog_stats():
    return {
        "hot_dogs_eaten": HotDog.objects.filter(
            state=HotDogState.EATEN,
        ).count(),
    }

The accompanying template, includes/hot-dog-stats.html, renders an HTML fragment with the context returned by the function:

<strong class=marquee>{{ hot_dogs_eaten }} dogs served!</strong>

Every template that needs the hot dog stats fragment can then render it with the inclusion tag like so:

{% load hot_dog_tags %}

...
<nav>
  ...
  {% hot_dog_stats %}
</nav>

This method is great for isolated pieces of content. But it does have the downside that the included template doesn’t have access to the outer context, so you may need to pass many variables through (see takes_context in the docs).

An ounce of prevention…

To be sure that none of your context processors do run database queries, now or in the future, you can add a test. Here’s a test case you should be able to paste into your project, in a file like example/tests/test_templates.py:

from django.db import connection
from django.db.utils import DatabaseError
from django.template import RequestContext
from django.template import Template
from django.test import RequestFactory
from django.test import TestCase


class TemplateTests(TestCase):
    request_factory = RequestFactory()

    def test_context_managers_no_queries(self):
        # Context managers do not make database queries to be performant and
        # robust
        # https://adamj.eu/tech/2023/03/23/django-context-processors-database-queries/
        template = Template("")
        context = RequestContext(self.request_factory.get("/"))

        def blocker(execute, sql, params, many, context):
            raise DatabaseError("Simulated broken connection 😱")

        with connection.execute_wrapper(blocker):
            result = template.render(context)

        assert result == ""

The test uses Django’s database instrumentation to simulate a broken database connection. The blocker() function will run during any query on the default database connection, and raise a DatabaseError rather than allow the query to run. So if a context processor does run a query, the test fail like so:

======================================================================
ERROR: test_context_managers_no_queries (example.tests.test_templates.TemplateTests.test_context_managers_no_queries)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../example/tests/test_templates.py", line 22, in test_context_managers_no_queries
    result = template.render(context)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  ...
  File "/.../example/context_processors.py", line 8, in hot_dog_stats
    "hot_dogs_eaten": HotDog.objects.filter(
  ...
  File "/.../example/tests/test_templates.py", line 19, in blocker
    raise DatabaseError("Simulated broken connection 😱")
django.db.utils.DatabaseError: Simulated broken connection 😱

Brilliant.

If your project has a custom error view, it’s also worth testing that the view works when queries don’t:

from http import HTTPStatus

from django.db import connection
from django.db.utils import DatabaseError
from django.test import RequestFactory
from django.test import TestCase

from example.views import server_error


class ErrorViewTests(TestCase):
    request_factory = RequestFactory()

    def test_server_error(self):
        # server_error runs for all errors, including database errors, so it
        # should work with a non-functional database connection
        # https://adamj.eu/tech/2023/03/23/django-context-processors-database-queries/
        def blocker(execute, sql, params, many, context):
            raise DatabaseError("Simulated broken connection 😱")

        # The view may also run before middleware, so avoid the test client
        # and call the view with a plain request without extra attributes
        # like 'request.user'
        request = self.request_factory.get("/")

        with connection.execute_wrapper(blocker):
            response = server_error(request)

        assert response.status_code == HTTPStatus.INTERNAL_SERVER_ERROR
        assert "error" in response.content.decode()

Test all the things!

Fin

This post is based on work for my client Silvr. Thanks to Florian Perucki, Gregory Tappero, Julien Lopinto, and Pascal Fouque for reviewing the relevant pull requests.

I hope you have found this post easy to process, at least in context,

—Adam

Learn how to make your tests run quickly in my book Speed Up Your Django Tests.

Subscribe via RSS, Twitter, Mastodon, or email:

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: