How to Set Up Source Maps with Django

2022-01-26 “Now where on this map is node_modules?”

Source maps are files that map your minified CSS or JavaScript back to the original code. They allow you to use your browser’s development tools to debug minified code as if it were the original. Also some error capture tools, such as Sentry, can use source maps to report errors for the original code.

In this post we’ll look at how source maps fit into Django’s static files infrastructure, including a couple recent changes I made to Django to better support them. We’ll look at using a JavaScript source map, but everything applies equally to CSS source maps. Alrighty, let’s dig in.

Directory Layout

Let’s look at an example project using source maps to see how they fit into Django’s infrastructure. This project demonstrates a static file setup that works well for most projects.

The project has three directories for “static files” with different roles, all in the root of repository:

frontend/, which contains the source CSS and JavaScript files. Django doesn’t use this directory at all. Frontend tools, like bundlers, should process the source files in frontend/ and place output files in the next directory, static/. This provides a clear “hand off”.
static/, which contains static files for Django to manage. This is in the STATICFILES_DIRS setting:
```
STATICFILES_DIRS = [BASE_DIR / "static"]
```
static_root/, which is created by Django’s collectstatic command. This is referred to by the STATIC_ROOT setting:
```
STATIC_ROOT = BASE_DIR / "static_root"
```
For non-development environments, you run collectstatic to populate this directory with static files from STATICFILES_DIRS and those inside apps. The static files storage can also process the files during this stage.

Overall the static file pipeline looks like:

+-----------+               +---------+                       +--------------+
| frontend/ | --- build --> | static/ | --- collectstatic --> | static_root/ |
+-----------+               +---------+                       +--------------+

Fan-tabby-tastic.

Build with Source Maps

There are many frontend build tools that can generate source maps, such as Webpack and Parcel. The example project keeps it simple with esbuild, a very fast and low configuration tool.

The example application code, in frontend/app.js, is minimal, only handling entering and exiting fullscreen mode:

document.addEventListener("keydown", (event) => {
  if (event.key == "f") {
    document.documentElement.requestFullscreen();
  } else if (event.key == "Escape") {
    document.exitFullscreen();
  }
});

esbuild minifies the code and outputs as a source map via an npm script in package.json:

{
  "dependencies": {
    "esbuild": "^0.14.12"
  },
  "scripts": {
    "build": "esbuild --minify --sourcemap frontend/app.js --outdir=static/"
  }
}

…which is invoked like so:

$ npm run build

> build
> esbuild --minify --sourcemap frontend/app.js --outdir=static/


  static/app.js      174b
  static/app.js.map  428b

⚡ Done in 10ms

The two output files are the minifiied code, app.js, and the source map, app.js.map. Here’s app.js:

document.addEventListener("keydown",e=>{e.key=="f"?document.documentElement.requestFullscreen():e.key=="Escape"&&document.exitFullscreen()});
//# sourceMappingURL=app.js.map

The second line is the source map reference, which is a specially formatted comment (as per the source map specification). When you open your browser's development tools, it will load source maps from these comments, and then you can debug your original source code.

A source map is a JSON file with a few bits of data. Here’s app.js.map:

{
  "version": 3,
  "sources": ["../frontend/app.js"],
  "sourcesContent": ["document.addEventListener(\"keydown\", (event) => {\n  if (event.key == \"f\") {\n    document.documentElement.requestFullscreen();\n  } else if (event.key == \"Escape\") {\n    document.exitFullscreen();\n  }\n});\n"],
  "mappings": "AAAA,SAAS,iBAAiB,UAAW,AAAC,GAAU,CAC9C,AAAI,EAAM,KAAO,IACf,SAAS,gBAAgB,oBAChB,EAAM,KAAO,UACtB,SAAS",
  "names": []
}

You can see:

version is the source map version.
sources contains original source file names.
sourcesContent contains each file’s original source code.
mappings contains compressed references that map the output code back to the source files.

Since source maps contain full original source code and more, they can become quite large. This is not a performance concern though, as they are only loaded when the development tools are open.

Source Maps in Development

In development, runserver directly serves static files from your static directories - here, static/. This means the source map reference works as expected. In Firefox, you can see this in action when you open the debugger, and it shows the source file:

Firefox development tools showing JavaScript source map in action

You can debug the minified code running in the browser as if it were the original source code. For example, if you set a breakpoint in the source file, it will pause the running code at the appropriate point. Amazing!

Additionally, stack traces will use the source file names, line numbers, and column numbers. If you add a delibarate error:

 document.addEventListener("keydown", (event) => {
   if (event.key == "f") {
     document.documentElement.requestFullscreen();
   } else if (event.key == "Escape") {
     document.exitFullscreen();
   }
+  throw "whoops";
 });

Then Firefox will report the problem at app.js, line 7, column 8:

Uncaught whoops 2 app.js:7:8
    <anonymous> app.js:7
    (Async: EventListener.handleEvent)
    <anonymous> app.js:1

That’s great, since it’s really hard to read the minified version’s single line 😅

Source Maps in Production

Source maps are really useful to have in your production environment, since they make it way easier to track down bugs. So it’s a good idea to ensure they are included in your static files.

(Some argue against shipping source maps to production for security, or to prevent people copying your code. I don’t buy either reason—“unminifier” tools already allow people to read your code without a source map.)

In production, Django’s collectstatic step will process your source map along with your other static files. What this does depends on the storage class selected in your STATICFILES_STORAGE setting.

If you’re using the default storage class, StaticFilesStorage, then you don’t need to do anything. The source map will be served as-is, alongside your other files, and all will work well. But StaticFilesStorage is not generally useful in production, since it is not compatible with setting caching headers on static files.

For most production setups, you’ll want to use Django’s ManifestStaticFilesStorage, which transforms files by appending a short hash. This implements the cache-busting pattern, where if the content of a file changes, its filename changes. You probably want to use ManifestStaticFilesStorage itself, or its subclass from Whitenoise, called CompressedManifestStaticFilesStorage. (Whitenoise makes it really easy to set up static files, by serving them direct from Django. I think it’s a good fit for most projects.)

Here’s what using collectstatic with Whitenoise looks like:

$ ./manage.py collectstatic -v 2
Copying '/Users/chainz/tmp/source-maps/static/app.js.map'
Copying '/Users/chainz/tmp/source-maps/static/app.js'
Post-processed 'app.js.map' as 'app.js.8dee7203977f.map'
Post-processed 'app.js' as 'app.ba8df8fcb9ec.js'
Post-processed 'app.js.8dee7203977f.map' as 'app.js.8dee7203977f.map.br'
Post-processed 'app.js.8dee7203977f.map' as 'app.js.8dee7203977f.map.gz'
Post-processed 'app.ba8df8fcb9ec.js' as 'app.ba8df8fcb9ec.js.br'
Post-processed 'app.ba8df8fcb9ec.js' as 'app.ba8df8fcb9ec.js.gz'

2 static files copied to '/Users/chainz/tmp/source-maps/static_root', 6 post-processed.

The hashing step renames app.js to app.ba8df8fcb9ec.js, and app.js.map to app.js.8dee7203977f.map. This is good for cache-busting, but it breaks the source map relation! Recall the comment:

//# sourceMappingURL=app.js.map

…it needs updating to use the renamed source map:

//# sourceMappingURL=app.js.8dee7203977f.map

(Without this update, the source map might still work. Vanilla ManifestStaticFilesStorage, and Whitenoise without WHITENOISE_KEEP_ONLY_HASHED_FILES on, both leave the unhashed files in place, which can allow the source map to load. But this will be without cache-busting, so you might get an old source map, and thus debug old source code - mega confusing!)

Thankfully, ManifestStaticFilesStorage can update these source map references for you. As part of the hashing step, it applies some regular expression based fixes to update file references, for example url() calls in CSS files. I contributed some extra fixes for source map references:

JavaScript source maps in Django 4.0. (Ticket #32383, thanks to Carlton Gibson and Mariusz Felisiak for the review.)
CSS source maps in Django 4.1 (expected August 2022). (Ticket #33446, thanks to Mariusz Felisiak for the review.)

On older Django versions, you can use the backport in the below section.

With either the backport or an updated Django version, you can see source map references get updated. For example, app.ba8df8fcb9ec.js contains:

document.addEventListener("keydown",e=>{throw e.key=="f"?document.documentElement.requestFullscreen():e.key=="Escape"&&document.exitFullscreen(),"whoops"});
//# sourceMappingURL=app.js.8dee7203977f.map

Brilliant. Now you can debug away those production woes!

Backport to the Future

Update (2022-03-03): I discovered that CSS source maps have more flexible whitespace requirements, so I’ve made a PR to Django and updated the below CSS source map regular expressions.

Here’s the backport for older Django versions. Place the class in your project and update your STATICFILES_STORAGE setting to use it, or merge with your existing custom storage class. This version is for Whitenoise - if you use plain ManifestStaticFilesStorage instead, make that the base class as appropriate.

import django

from whitenoise.storage import CompressedManifestStaticFilesStorage


class CustomStaticFilesStorage(CompressedManifestStaticFilesStorage):
    if django.VERSION < (4, 0):
        patterns = CompressedManifestStaticFilesStorage.patterns + (
            (
                "*.js",
                (
                    (
                        r"(?m)^(//# (?-i:sourceMappingURL)=(.*))$",
                        "//# sourceMappingURL=%s",
                    ),
                ),
            ),
            (
                "*.css",
                (
                    (
                        r"(?m)^(/\*#[ \t](?-i:sourceMappingURL)=(.*)[ \t]*\*/)$",
                        "/*# sourceMappingURL=%s */",
                    ),
                ),
            ),
        )
    elif django.VERSION < (4, 1):
        # Django 4.0 switched to named patterns
        patterns = CompressedManifestStaticFilesStorage.patterns + (
            (
                "*.css",
                (
                    (
                        r"(?m)^(?P<matched>/\*#[ \t](?-i:sourceMappingURL)=(?P<url>.*)[ \t]*\*/)$",
                        "/*# sourceMappingURL=%(url)s */",
                    ),
                ),
            ),
        )
    else:
        raise AssertionError(
            "The above backported custom patterns are no longer required."
        )

After Django 4.1, the backport isn’t required, so it raises an AssertionError to remind you to remove it. I like to do this to ensure things get tidied up.

Fin

May source maps guide your path,

—Adam

Learn how to make your tests run quickly in my book Speed Up Your Django Tests.

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: django