GitHub Actions: Faster Python runs with cached virtual environments

2023-11-02 Kiwi go any faster?

Most projects I work on use Python, good ol’ Pip, and pip-tools. Below is a pattern I’ve used to speed up the GitHub Actions workflow runs on several such projects. On larger projects with many dependencies, it can save tens of seconds per run.

# ...

jobs:
  example:
    # ...

    steps:
      # ...

      - uses: actions/setup-python@v4
        id: setup_python
        with:
          python-version: '3.12'

      - name: Cache virtualenv
        uses: actions/cache@v3
        with:
          key: venv-${{ runner.os }}-${{ steps.setup_python.outputs.python-version}}-${{ hashFiles('requirements.txt') }}
          path: .venv

      - name: Install dependencies
        run: |
          python -m venv .venv
          source .venv/bin/activate
          python -m pip install -r requirements.txt
          echo "$VIRTUAL_ENV/bin" >> $GITHUB_PATH
          echo "VIRTUAL_ENV=$VIRTUAL_ENV" >> $GITHUB_ENV

      # ...

The steps here:

Set up Python with actions/setup-python.
Cache the .venv directory with actions/cache.

The cache key combines the operating system, full Python version, and the hash of requirements.txt. (requirements.txt pins all versions, as generated by pip-tools.)

In most runs, this step restores the whole virtual environment. But if any of those inputs change, the virtual environment will be built from scratch and later saved.
Run commands to set up the virtual environment, install dependencies, and activate it for all the following steps.

python -m venv is a no-op if the cache step restored the virtual environment since it already exists. Similarly, pip install doesn’t take long in restored environments, as it only verifies that the packages are installed (“Requirement already satisfied”).

The echo commands effectively “activate” the virtual environment for all further steps by adding the path and VIRTUAL_ENV environment variable. Later steps can use installed packages without any reference to .venv.

Contrast with the below typical approach, as recommended by the actions/setup-python documentation:

# ...

jobs:
  example:
    # ...

    steps:
      # ...

      - uses: actions/setup-python@v4
        with:
          python-version: '3.12'
          cache: pip

      - name: Install Python dependencies
        run: python -m pip install -r requirements.txt

      # ...

The cache: pip line causes setup-python to save and restore Pip’s wheel cache (as managed with pip cache). This cache makes pip install somewhat faster, as Pip doesn’t need to download dependencies or build them into wheels. But it doesn’t make it instant, as each wheel still needs installing.

Fin

May your actions run ever faster,

—Adam

Learn how to make your tests run quickly in my book Speed Up Your Django Tests.

One summary email a week, no spam, I pinky promise.

Related posts:

Tags: github, python