How to Use Django’s Parallel Testing on macOS With Python 3.8+
Django’s test
command takes the --parallel
flag to activate parallel testing. Unfortunately, this has never worked on Windows, and with Python 3.8 it has stopped working on macOS.
Django uses the “fork” method in multiprocessing
to start the new test processes. Windows does not support “fork”, and whilst macOS does, Python 3.8 changed its default to “spawn”. If the start method is not “fork” when you run tests with the --parallel
flag, Django silently switches back to serial testing.
The “spawn” method works quite differently to “fork”, requiring a lot of initialization in the new test processes, so the current test framework is incompatible. This is a known issue, Ticket #31169, and it is being worked on by Ahmad A. Hussein as part of Django’s Google Summer of Code 2020. Hopefully we will merge the fix in Django 3.2.
For the time being, there’s a workaround for macOS. Alternatively, you can switch to pytest, since its parallelization plugin, pytest-xdist, works on all platforms.
On macOS on Python 3.8+, you can work around the lack of “spawn” support by reverting the multiprocessing start method to “fork”. This is only a temporary workaround, and has a small risk of introducing system library crashes to your test suite. Python 3.8 changed the macOS default because many system libraries crash when used after forking (Python Issue #33725).
The first step in the workaround is to disable the macOS system protection against forking. Do this by setting this environment variable in the terminal you run tests:
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
You can do this in your shell initialization file so it always applies - see this Stack Overflow answer.
The second step is to add a call to multiprocessing.set_start_method()
at the top of main()
in your manage.py
file. This is best accompanied with a check that OBJC_DISABLE_INITIALIZE_FORK_SAFETY
has indeed been set:
#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import multiprocessing
import os
import sys
def main():
try:
command = sys.argv[1]
except IndexError:
command = "help"
if command == "test" and sys.platform == "darwin":
# Workaround for https://code.djangoproject.com/ticket/31169
if os.environ.get("OBJC_DISABLE_INITIALIZE_FORK_SAFETY", "") != "YES":
print(
(
"Set OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES in your"
+ " environment to work around use of forking in Django's"
+ " test runner."
),
file=sys.stderr,
)
sys.exit(1)
multiprocessing.set_start_method("fork")
...
The check will stop manage.py test
from starting on macOS if OBJC_DISABLE_INITIALIZE_FORK_SAFETY
has not been set.
Read my book Boost Your Git DX to Git better.
One summary email a week, no spam, I pinky promise.
Related posts:
- Make Django Tests Always Rebuild the Database if It Exists
- Getting a Django Application to 100% Test Coverage
- Avoid Hardcoding IDs in Your Tests
Tags: django