Using Sharding with Playwright

Published in

DevOps.dev

8 min readMar 1, 2024

Hello everyone, in this brief article we will explore how can we use sharding feature in Playwright to distribute our tests across various shards in order to speed up our test executions and reduce the overall feedback time in our CI process. Let’s get started, but first I would like to provide some information about parallelism in Playwright.

Parallelism

In playwright, we have provision to run the tests in parallel and it is set to true by default in the playwright.config.js file. With this enabled, it runs several worker processes that run at the same time. Tests in a single file are run in order, in the same worker process.

Also parallel tests are executed in separate worker processes and cannot share any state or global variables. There are also quite a few ways to configure tests to be run in parallel which are mentioned below.

Using test.describe.configure({ mode: ‘parallel’ });
We can use fullyParallel: true to enable parallelism for all the tests
Or we can enable fullyParallel: true for some projects say chromium, firefoxe, webkit etc.

To disable parallelism, we can configure tests to be run using just 1 worker, we can do so via config file by setting the workers to 1 or pass the same from command line while using npx playwright test — workers=1.

For more information about parallelism, you can refer this page.

What is Sharding?

The word Sharding originates from a database architecture concept, wherein we break a large database into multiple chunks or shards in order to distribute the load, optimize the workload, improve efficiency and scalability of the system. This is somewhat similar to horizontal scaling and each shard behaves as an independent unit with its own resources and manages data accordingly to improve query performances and eliminate single points of failures while using a standalone database server which might handle huge amounts of data.

You can read more about sharding here, additionally Gaurav Sen has a amazing video explaining this, refer video.

Now, coming to Playwright world, here the concept of sharding can be used to improve the overall test execution times by splitting the number of tests across various shards or multiple machines simultaneously. We can do so using the shard parameter i.e shard=x/y where x is the shard index and y is the total number of shards.

npx playwright test --shard=1/4
npx playwright test --shard=2/4
npx playwright test --shard=3/4
npx playwright test --shard=4/4

Here, we have divided our total tests into 4 shards hence the configured number of workers will then work on each of these shards to execute the total number of tests and reduce the overall test execution time significantly.

For demo purposes, I have installed playwright in my local machine and will be using the sample boilerplate code provide in the tests-examples folder wherein we have some tests that are configured to run on the playwright to-do mvc website (URL: https://demo.playwright.dev/todomvc) Here we have around 24 tests which runs on 3 browsers so overall 72 tests for a single file, have copied same file twice so with 3 spec files we have 216 tests across 3 browsers — chromium, firefox and webkit. I have renamed the spec files accordingly as seen below.

In the playwright.config.js I have set the below configurations, setting parallel to true and workers as 6.

fullyParallel: true

workers: process.env.CI ? 6 : 6

This implies that when I run the command: npx playwright test all the 216 tests will be run using 6 workers in my local and CI. Below tests run on local machine using 6 workers.

Now, if I run command npx playwright test along with the shard parameter specifying the total shards as 4, firstly the 216 tests will be divided into 4 chunks each with 54 tests. These 54 tests will then be executed using 6 workers in shard 1, 54 tests with 6 workers in shard 2 as seen below. As each shard will be executed on its own agent machine the level of parallelism is much deeper and our tests will be executed significantly faster rather than just using 6 workers which is default configuration setup.

But with sharding we encounter an hiccup with reporting with default html reporter option. Post test execution, when we run the command - ‘npx playwright show-report’ we will see that the generated report will only show details of the last run shard with just 54 tests as seen below. But what about the remaining tests which had been executed in previous shards.

To solve this, we will need to configure the reporter as ‘blob’ when we are using sharding. By default each shard will generate its own blob reports in a folder called blob-report we then will have to merge all these individual blob-reports from all the shards to get a consolidated report for all the tests that have been executed across shards.

export default defineConfig({
  reporter: process.env.CI ? 'blob' : 'html'
});

To merge reports from multiple shards, put the blob report files into a single directory, for example in our case all-blob-reports,then finally HTML report will be available in playwright-report folder.

npx playwright merge-reports --reporter html ./all-blob-reports

Sharding in CI using Github Actions

Now, in order to effectively use sharding we can leverage CI tools such as Github Actions wherein we have provision to easily setup and run tests on multiple agent machines. To do so we need to include the following points in our Github actions workflow file.

Adding a strategy with matrix option to the job with shardIndex and shardTotal values.
Run the npx playwright test command specifying the shardindex along with shardTotal options.
Upload the individual shard blob report to Github actions workflow artifacts so that merge-report job can pickup this up and generate the consolidate HTML report.

name: Sharding Tests
on:
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]

jobs:
  run-tests:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4, 5, 6 ]
        shardTotal: [6]
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
      with:
        node-version: 20
    
    - name: Install dependencies
      run: npm ci

    - name: Install Playwright Browsers
      run: npx playwright install --with-deps

    - name: Run Playwright tests
      run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
    
    - name: Upload blob report to GitHub Actions Artifacts
      if: always()
      uses: actions/upload-artifact@v4
      with:
        name: blob-report-${{ matrix.shardIndex }}
        path: blob-report
        retention-days: 1

For job “run-tests” we will be using a total of 6 shards specified under matrix with fail-fast set to false. Then we will execute npx playwright test using the specific shardIndex and shardTotal value for example 1/6, 2/6 etc. Finally once the shards are executed, the individual shard blob-report will be uploaded to actions artifacts under blob-report with shardIndex suffix.

Next, we will need to add a “merge-reports” job which will depend on the run-tests job, what this job will do is download all the individual shard blob-report into a specified folder called all-blob-reports and then run the merge-reports command to get a combined HTML report and upload it the GitHub actions Artifacts tab.

merge-reports:             
      if: always()
      needs: [run-tests]
          
      runs-on: ubuntu-latest
      steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 18
      - name: Install dependencies
        run: npm ci
          
      - name: Download blob reports from GitHub Actions Artifacts
        uses: actions/download-artifact@v4
        with:
          path: all-blob-reports 
          pattern: blob-report-*
          merge-multiple: true
          
      - name: Merge into HTML Report
        run: npx playwright merge-reports --reporter html ./all-blob-reports 
          
      - name: Upload HTML report
        uses: actions/upload-artifact@v4
        with:
          name: html-report--attempt-${{ github.run_attempt }}
          path: playwright-report
          retention-days: 7

This is my full worflow file along with both the jobs

name: Playwright tests using sharding
on:
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]
jobs:
  run-tests:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4, 5, 6 ]
        shardTotal: [6]
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
      with:
        node-version: 20
    - name: Install dependencies
      run: npm ci
    - name: Install Playwright Browsers
      run: npx playwright install --with-deps
    - name: Run Playwright tests
      run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
    
    - name: Upload blob report to GitHub Actions Artifacts
      if: always()
      uses: actions/upload-artifact@v4
      with:
        name: blob-report-${{ matrix.shardIndex }}
        path: blob-report
        retention-days: 1

  merge-reports:             
      if: always()
      needs: [run-tests]
          
      runs-on: ubuntu-latest
      steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 18
      - name: Install dependencies
        run: npm ci
          
      - name: Download blob reports from GitHub Actions Artifacts
        uses: actions/download-artifact@v4
        with:
          path: all-blob-reports 
          pattern: blob-report-*
          merge-multiple: true
          
      - name: Merge into HTML Report
        run: npx playwright merge-reports --reporter html ./all-blob-reports 
          
      - name: Upload HTML report
        uses: actions/upload-artifact@v4
        with:
          name: html-report--attempt-${{ github.run_attempt }}
          path: playwright-report
          retention-days: 7

Once the workflow file is pushed to Github, all the total tests: 216 will be distributed and run on specific shard machine in our case 6 machines. As seen below 36 tests are executed in run-tests (1,6) machine. Similarly 36 tests will be executed using 6 workers in other shard agent machine. Once all the tests are executed the merge report job will be triggered and then generate our consolidated HTML report.