Why we chose Playwright over Cypress
The Taming of the Testing: A modern Shakespearean tragedy
I recently had the opportunity to work with a development team on a greenfield project. The project consisted of a Single Page Application with a few APIs to display features on a map and download associated files. As it was going to be all shiny and new we thought “great, let’s use Cypress it should do everything we need”, namely:
- Automatic waiting so no need to build a substantial framework to handle this
- Mocks network requests to enable us to solely test our UI
- Ability to take screenshots and videos of test suites
- A Docker container for running the tests in Azure DevOps
Unfortunately, it didn’t quite go to plan. Here’s the full story of our journey…
At the UKHO we have been trying to branch out and try new things when it comes to test automation. We all too often get stuck in the same rut of using Selenium and the associated vast frameworks and flaky tests. These typically rely heavily on intertwined test data and external systems, that most of the time we don’t need to worry about about when functionally testing the application we are developing.
It was all going rather too well with Cypress…
The team was enjoying testing, I mean why wouldn’t they anyway, and here are a few reasons why:
- Cypress was very intuitive
- super easy to mock requests to use static test data
- easy to get the tests running in the docker container in out Azure DevOps pipeline
- simple enough to run the tests across multiple environments and cross browser
- the test runs were quick and even quicker in the pipelines
…and then we implemented Microsoft Authentication Library for JavaScript (MSAL.js) using Azure AD B2C.
We knew Cypress couldn’t handle cross-origin windows (or in fact shouldn’t really be used across multiple windows / tabs anyway as Cypress should only be testing your application) but we thought “that’s fine” as they provide example recipes to handle Single Sign On using the API and cy.request().
Our implementation requires interactive login, so we couldn’t use the API to store the required auth credentials in the local session for each test.
Sad, as this would have been nice and clean, but it’s fine (or so we thought) there’s another solution doing the rounds on the forums of using Puppeteer to take control of the Cypress window and login, the session is logged in and we should then be able to carry on with Cypress.
Nope, this won’t work for us either… after spending a good chunk of time trying to get it running, we could not get it running consistently and as it was such a ‘dirty’ workaround we abandoned this approach as no one had much confidence in it.
After calmly spending some time trying to salvage our beloved Cypress tests, we eventually had to concede and look at either rewriting our tests or introducing a secondary tool to cover the e2e tests. So, at this point you are probably wondering…
“why couldn’t you stay with Cypress for testing our developed UI and use another tool (probably Selenium) for your e2e / integration testing”
Well here are just a few of the reasons:
- stubbing out the authorisation and payment aspects of the application: extra maintenance, time consuming, we wouldn’t be able to test a deployed instance of the application, stubbing also would move us further away from a true end-to-end test
- maintaining two frameworks
- it takes time to get Selenium right
- the associated overhead with Selenium (drivers, frameworks, separate test project)
How did we get over this?
We felt slightly uncomfortable with Puppeteer after our previous experiences with it:
- not as intuitive as Cypress
- the need for additional waits (e.g. on click it does not implicitly wait)
- lack of cross browser support
And then we stumbled upon Playwright…
- members of the Google Puppeteer team moved to Microsoft
- the APIs are near identical, but they were able to make improvements that essentially wouldn’t be possible in Puppeteer
- it is used to test VS Code web builds and typescript.js across browsers (to name a few)
It seemed like it would do everything we wanted and needed… so we spiked it.
Playwright saves the day
Playwright managed to provide us everything we needed and we liked it!
What do we like about Playwright:
- better cross browser support than Cypress (Chromium, Firefox and WebKit)
- easy to intercept network activity (stubbing / mocking / awaiting)
- in built waits on most actions
- support of multi-page and domains
- we could use the same test runner as our unit tests
And that is that, on that specific project we are now using Playwright, but that certainly does not mean we won’t use Cypress in the future. If the application is truly an SPA then Cypress is still the best and go-to tool for us.
Some things we learnt & some we already knew
- It’s good to celebrate and learn from failure, innovation should not be discouraged
- Cypress is GOOD, very good– it just wasn’t quite right for us ☹
- We wanted to keep it SIMPLE and utilise ONE tool for UI and e2e testing — some may disagree with this
- Selenium isn’t BAD — it just wasn’t quite right for us either or what we wanted
- We tried Puppeteer, but it just felt a bit ‘clunky’ and not as intuitive as Cypress or Playwright
- Playwright fills some of the Cypress gaps: multiple pages, popup windows, third party sites, better browser support
- We will add both Playwright and Cypress into our Test Tooling arsenal for UI and e2e, alongside the already tried and tested Selenium
