GitHub Actions: Faster Python runs with cached virtual environments

2023-11-02 Kiwi go any faster?

Update (2024-07-19): Updated from the single cache action to use the separate cache/restore and cache/save steps, saving the virtual environment before running tests. This change ensures the virtual environment can be reused even when tests fail.

Most projects I work on use Python, good ol’ Pip, and pip-tools. Below is a pattern I’ve used to speed up the GitHub Actions workflow runs on several such projects. On larger projects with many dependencies, it can save tens of seconds per run.

# ...

jobs:
  example:
    # ...

    steps:
      # ...

      - uses: actions/setup-python@v5
        id: setup_python
        with:
          python-version: '3.12'

      - name: Restore cached virtualenv
        uses: actions/cache/restore@v4
        with:
          key: venv-${{ runner.os }}-${{ steps.setup_python.outputs.python-version }}-${{ hashFiles('requirements.txt') }}
          path: .venv

      - name: Install dependencies
        run: |
          python -m venv .venv
          source .venv/bin/activate
          python -m pip install -r requirements.txt
          echo "$VIRTUAL_ENV/bin" >> $GITHUB_PATH
          echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> $GITHUB_ENV

      - name: Saved cached virtualenv
        uses: actions/cache/save@v4
        with:
          key: venv-${{ runner.os }}-${{ steps.setup_python.outputs.python-version }}-${{ hashFiles('requirements.txt') }}
          path: .venv

      # ...

The steps here:

Set up Python with actions/setup-python.
Attempt to restore a cached virtual environment with actions/cache/restore (a sub-action of the typical actions/cache).

The cache key combines the operating system, full Python version, and the hash of requirements.txt. (requirements.txt pins all versions, as generated by pip-tools.)

In most runs, this step restores the whole virtual environment. But if any of those inputs change, the virtual environment will be built from scratch and saved later.
Run commands to set up the virtual environment, install dependencies, and activate it for all the following steps.

python -m venv is a no-op if the cache step restored the virtual environment since it already exists. Similarly, pip install doesn’t take long in restored environments, as it only verifies that the packages are installed (“Requirement already satisfied”).

The echo commands effectively “activate” the virtual environment for all further steps by adding the path and VIRTUAL_ENV environment variable. Later steps can use installed packages without any reference to .venv.
Save the virtual environment to the cache with actions/cache/save.

This happens before running tests, so that even if they fail, the cached virtual environment can be reused on subsequent runs. This is useful in many situations, particularly if a new dependency is causing the failure. (The default behaviour when using the non-separated actions/cache is to save only on success, repeating work when there are failures.)

Contrast with the below typical approach, as recommended by the actions/setup-python documentation:

# ...

jobs:
  example:
    # ...

    steps:
      # ...

      - uses: actions/setup-python@v4
        with:
          python-version: '3.12'
          cache: pip

      - name: Install Python dependencies
        run: python -m pip install -r requirements.txt

      # ...

The cache: pip line causes setup-python to save and restore Pip’s wheel cache (as managed with pip cache). This cache makes pip install somewhat faster, as Pip doesn’t need to download dependencies or build them into wheels. But it doesn’t make it instant, as each wheel still needs installing.

Fin

May your actions run ever faster,

—Adam

Learn how to make your tests run quickly in my book Speed Up Your Django Tests.

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: github, python