I started using Cypress back in 2019 when the company I worked for decided to move away from Protractor for new projects. At that time, Angular was the framework I was using, and I got the opportunity to implement the Cypress PoC. Recently, I switched jobs and am now working with React, where I also got the chance to implement the Playwright PoC.
Having experience with both Angular and React, the use of data-testid attributes for testing has been my implementation preference. This has allowed me to maintain a consistent approach to UI end-to-end testing, and I haven’t observed any significant differences in testing between Angular and React applications.
NOTE: I use a React app as an example for both Cypress and Playwright in this article.
What do both testing frameworks offer?
Both Cypress and Playwright offer a fantastic UI testing experience (you can also test APIs). They make it easy for developers to write tests because of their auto-wait functionality, they provide a UI to visualise your tests as they run, also generate screenshots and videos of your tests and support typescript.
Both frameworks also support visual component testing, but I will not be covering this topic in this article.
Since both frameworks offer very similar functionality, I will dissect how each achieves the following aspects and how they affect the developer experience/productivity:
- The syntax used to write tests plays a crucial role in the learning curve, and overall ease of use of each framework. Including but not limited to extending the framework with custom commands and recording tests.
- Test execution and maintainability are essential factors that impact the confidence developers have in their tests and the amount of time spent debugging them, particularly concerning speed and stability.
- Test reports play a critical role in the testing process, and evaluating their ease of setup and the level of information they provide is essential for both frameworks.
My Cypress Experience
Installing Cypress was easy enough, mostly just an NPM dependency and off you go. However, it did not take long to encounter the intricate details that you need to know about Cypress to be able to work with it. Here are some of the predominant ones:
- Syntax used to write tests: Cypress uses a promise-like syntax to write tests. This may not seem confusing at first, but developers tend to think that because their API looks like a promise, it behaves like a promise (async/await). The sad reality is that it does not, and this causes devs to spend a lot of time learning how to work with the Cypress API and make it fit their particular testing scenario.
If you need to work with async/await, you have 2 options: either wrap it in Cypress commands to be able to add it to the Cypress command chain, or use a library likecypress-promise. Here is an example of each option:
// async-await.spec.ts
import promisify from 'cypress-promise';
function sleep(milliseconds: number) {
return new Promise((resolve) => setTimeout(resolve, milliseconds));
}
async function asyncFunction(text: string) {
console.log('started asyncFunction ' + text);
await sleep(3000);
console.log('finalized asyncFunction ' + text);
}
context('Async/Await Test', () => {
beforeEach(() => {
cy.visit('/');
});
it('convert promises into cypress commands, do not write tests using async/await', () => {
cy.wrap(null).then(() => asyncFunction('first'));
cy.wrap(null).then(() => asyncFunction('second'));
});
it('convert cypress commands to promises, should be able to code with async/await', async () => {
const foo = await promisify(cy.wrap('foo'));
const bar = await promisify(cy.wrap('bar'));
expect(foo).to.equal('foo');
expect(bar).to.equal('bar');
});
});- Test Reports: Cypress lacks a built-in test report feature, which means obtaining a comprehensive summary of all tests, their durations, and additional elements like screenshots and linked videos for failures can be a challenging and cumbersome process. Despite investing considerable effort in configuring the reports, there might still be instances where screenshots are missing from the generated reports.
// package.json: example of cumbersome test reporting setup
"scripts": {
"-------------------- E2E Commands --------------------": "",
"cypress:open": "nyc cypress open",
"cypress:run": "npm-run-all -s --continue-on-error _clean _cypress-run:run _cypress-run:html",
"-------------------- Supporting Commands --------------------": "",
"_clean": "npx rimraf coverage cypress/output/junit cypress/output/mocha-json cypress/output/mocha-html/*.html cypress/output/mocha-html/*.json .nyc_output",
"_coverage-report": "npx nyc report --reporter html",
"_cypress-run:run": "nyc cypress run --headless --browser chrome",
"_cypress-run:html:merge-json": "mochawesome-merge cypress/output/mocha-json/*.json > cypress/output/mocha-html/merged-mochawesome.json",
"_cypress-run:html:gen-html-from-json": "marge cypress/output/mocha-html/merged-mochawesome.json -f cypress -o cypress/output/mocha-html -i true --charts true",
"_cypress-run:html": "npm-run-all -s _cypress-run:html:merge-json _cypress-run:html:gen-html-from-json _coverage-report"
},- How to run headed tests: Cypress runs your web-app inside of its own browser app. It first opens a page where it lists all of your
specs, you can then click on a spec to run all tests inside of that spec. I might add that all of this is quite slow. Furthermore, if you would like to only run one test inside of yourspec, you need to mark it asonly, save and then only that test will auto reload.
- How to access browser console logs in headless runs: Browser console logs are only visible when you run the tests in headed form (with DevTools open). This means that if your tests run successfully locally but fail on your CI pipeline, then you are in for a fun time trying to get those logs to the pipeline console. Luckily there is a plugin that facilitates this, but it was not just a plug-and-play job due to the way Cypress plugins work.
// cypress/plugins/index.ts
interface Browsers {
family: string;
name: string;
}
interface LaunchOptions {
args: string[];
}
const cypressPLugins = (on: unknown, config: unknown) => {
require('@cypress/code-coverage/task')(on, config);
// log console.* messages to the cypress console,
// it helps when there are errors on the CI/CD pipeline
require('cypress-log-to-output').install(on, consoleToLogConfig);
require('@cypress/react/plugins/react-scripts')(on, config);
// @ts-ignore
on('before:browser:launch', (browser: Browsers, launchOptions: LaunchOptions) => {
if (browser.family === 'chrome' || browser.name === 'chrome') {
console.log('Adding chrome config...');
launchOptions.args.push('--disable-dev-shm-usage');
launchOptions.args.push('--lang=en');
}
return launchOptions;
});
return config;
};
export const consoleToLogConfig = (_type: unknown, event: { level: string; type: string }) => {
// return true or false from this plugin to control if the event is logged on the cypress console
// `type` is either `console` or `browser`
// if `type` is `browser`, `event` is an object of the type `LogEntry`:
// https://chromedevtools.github.io/devtools-protocol/tot/Log/#type-LogEntry
// if `type` is `console`, `event` is an object of the type passed to `Runtime.consoleAPICalled`:
// https://chromedevtools.github.io/devtools-protocol/tot/Runtime/#event-consoleAPICalled
// only show error events:
return event.level === 'error' || event.type === 'error';
};
export default cypressPLugins;- Debugging Tests: There are a few different options to debug in Cypress and there are many articles about this topic out there. The main take away is that because Cypress commands run asynchronously, you cannot just add a
debuggercommand.
Essentially, Cypress provides you with 2 useful commands:debug()andpause(). The first one adds adebuggercommand to the Cypress runner ,while the later one stops your test run at that point. Both provide you with the ability to inspect the DOM, but only withpause()can you step through each of the upcoming Cypress commands in your test.
Given that Cypress uses a promise-like chaining API, you can chain either of those commands with any Cypress command:
cy.get([data-testid="username-input"]).type("my-username").pause();
- Test Parallelisation: Cypress does not support running tests in parallel locally. There are plenty of libraries out there to help you achieve parallelisation, however, this is no easy feat. In my previous company we tried to parallelise our tests in order to improve pipeline runtime, but the effort turned out to be so great that we de-prioritised it.
The side effect, is that developers rarely run the full Cypress tests suit locally, they just waited for feedback from the pipeline. - Speed: Cypress is slow when it runs both in headless and headed browser format. This is the main reason we wanted to parallelise tests. Even running tests locally is incredibly slow. When developing a test, the test auto-reloads when you modify the file, this can take a couple of seconds even on my MacBook Pro.
- Tests Stability: Cypress is known for being flaky, mostly on CI/CD pipelines and it causes devs to spend a lot of time chasing ghosts. The usual hacky solution I have seen devs implement, is to add a
cy.wait(<ms>)to the test and adding a retry to the CI/CD pipeline. - Custom Commands: If you have common functionality that you use often in your tests, you will most likely want to create a custom Cypress command for it. Unfortunately, adding such commands is not intuitive nor type-safe (unless you go the extra mile):
// cypress/support/commands.ts
Cypress.Commands.add('getByTestId', (selector: string, ...args: unknown[]) => {
return cy.get(`[data-testid=${selector}]`, ...args)
});
// cypress/typings/cypress.d.ts
declare namespace Cypress {
interface Chainable {
getByTestId(selector: string, ...args: unknown[]): Chainable;
}
}
// you can now use this command in your test:
it('should type username in username input', () => {
cy.getByTestId('username-input').type('my-username');
});- Recording Tests: Cypress now offers the Cypress Studio, which is currently an experimental feature that I have personally not tried yet.
My Playwright Experience
I started working with React at the start of 2023 and my team did not have any UI tests yet. Therefore I decided to look into introducing Cypress. However, we had also started introducing web-components and we planned to use Storybook to document and test our web-components. Since Storybook uses Playwright under the hood I investigated what Playwright could offer. The initial appeal was: one single framework that my team would have to learn to test both web-components and our UI.
Installing Playwright was a walk in the park: I had full tests running in less than an hour.
- Syntax used to write tests: Writing Playwright tests is as simple as writing normal typescript code, there is no special API to learn. The best part is that you can write normal async/await code so you can use all of your normal typescript supporting functions. Example:
import { test } from '@playwright/test';
test.describe('Playwright test with async/await', () => {
function sleep(milliseconds: number) {
return new Promise((resolve) => setTimeout(resolve, milliseconds));
}
async function asyncFunction(text: string) {
console.log('started asyncFunction ' + text);
await sleep(3000);
console.log('finalized asyncFunction ' + text);
}
test('should open storybook and click around', async ({ page }) => {
await page.goto('http://localhost:6006/');
await page
.getByRole('link', {
name: 'Storybook 7.1.1 is available! Your current version is: 7.0.2 Dismiss notification',
})
.click();
await page.getByRole('link', { name: 'Storybook' }).click();
await page.locator('#internal-components-overview--docs').click();
});
test('should test the async methods used in previous example', async ({
page,
}) => {
await asyncFunction('first');
await asyncFunction('second');
});
});- Test reports: This was such a pleasant surprise: I did not have to do anything! The report that Playwright generates out-of-the-box is fantastic. You are also able to run your tests on multiple browsers very easily (just change the config, Playwright will install the browsers it needs for you).
I have it configured to only include trace-information, screenshots and videos for failing tests; otherwise, just like Cypress, test runs are slow. In the example below, our first test failed. If we click on it, we get the following info: the exception causing the failure, which steps of the test failed, a video of the test, and full trace of the test.
- How to run headed tests: Playwright is developed by Microsoft, so of course they provide a plugin for VS Code. I am an IntelliJ user myself, but I find myself switching to VS Code when I want to run Playwright tests.
Unlike Cypress, you dont have to start anything to be able to run one or multiple tests. Just open your code in VS Code and click on the usual unit test green triangle, or the green-tick/red-X if you have already run the test in the past. You can also choose if you would like the browser to open or the trace viewer.
The developer experience here is fantastic. Tests open fast and are very responsive.
- How to access browser console logs in headless runs: Like Cypress, Playwright does not show browser logs in the console where Playwright is running. In order to achieve this, you can extend the Playwright
Pageobject to forward browser logs to the console log (there are other ways of doing this, for example, extending thetestobject itself).
// utils/console.util.ts
export const configureLogForwarding = (page: Page) => {
page.on('console', (msg) => {
if (process.env.PLAYWRIGHT_LOG_TO_CONSOLE === 'true') {
switch (msg.type()) {
case 'info':
case 'log': {
// eslint-disable-next-line no-console
console.log(`Log: "${msg.text()}"`);
break;
}
case 'warning': {
// eslint-disable-next-line no-console
console.log(`Warning: "${msg.text()}"`);
break;
}
case 'assert':
case 'error': {
// eslint-disable-next-line no-console
console.log(`Error: "${msg.text()}"`);
break;
}
}
}
});
};
// tests/demo-tests.spec.ts
test.describe('Playwright test with async/await', () => {
test.beforeEach(async ({ page }) => {
configureLogForwarding(page);
hostAppNavigationPo = new HostAppNavigationPo(page);
genericAssetModalPo = new GenericAssetModalPo(page);
});
// ...- Debugging Tests: This is one of my favourite features. To debug a test, set a break-point and right-click on the green-triangle/green-tick/red-x and click on Debug Test. The browser will open and the run will stop at your break-point. From here on, it is a normal debugging session that devs are used to.
- Test Parallelisation: Playwright supports parallelisation out-of-the-box (your system under test must of course be able to support tests running in parallel). Given that our CI/CD tool does not have a lot of resources and is therefore slow, we have turned parallel testing off in our pipeline. However, when I am coding, I am able to run all of our tests locally very fast. Granted, we have only been using Playwright for about 5 months so we only have ~80 tests. Here are some non-scientific numbers: ~80 tests, 5m in CI/CD (1 worker), 2m locally (1 worker), 55s locally (5 workers).
// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
/* other configuration... */
/* Opt out of parallel tests on CI. */
workers: process.env.CI ? 1 : undefined,
/* Run tests in spec files in parallel */
fullyParallel: true,
});- Speed: When running equivalent tests in headless form (as you would in a CI/CD pipeline), Playwright is ~1.5 faster than Cypress.
Furthermore, Playwright is able to start tests on the browser in less than a second (on my MacBook Pro). This means that I can run any test, in any spec, at any time without having to start a Playwright server, etc. Unlike Cypress, for which you first have to start the Cypress UI and then navigate to the test you would like to run within that UI.
This makes for a fantastic developer experience! - Tests Stability: We have been writing tests for the last 5 months, and we are yet to experience any flakiness and have not had had to use any waits or sleeps in our tests. Since I am no longer working at the previous company where we had Cypress, it is difficult for me to ascertain if migrating those flaky tests to Playwright would stabilise them.
- Custom commands: Playwright offers a way of extending the base
testso that you can have custom commands and/or page objects easily accessible during your tests. This pattern is simple to follow, intuitive and type-safe. Example from Playwright:
// my-test.ts
import { test as base } from '@playwright/test';
import { TodoPage } from './todo-page';
export type Options = { defaultItem: string };
// Extend basic test by providing a "defaultItem" option and a "todoPage" fixture.
export const test = base.extend<Options & { todoPage: TodoPage }>({
// Define an option and provide a default value.
// We can later override it in the config.
defaultItem: ['Do stuff', { option: true }],
// Define a fixture. Note that it can use built-in fixture "page"
// and a new option "defaultItem".
todoPage: async ({ page, defaultItem }, use) => {
const todoPage = new TodoPage(page);
await todoPage.goto();
await todoPage.addToDo(defaultItem);
await use(todoPage);
await todoPage.removeAll();
},
});
// example.spec.ts
import { test } from './my-test';
test('test 1', async ({ todoPage }) => {
await todoPage.addToDo('my todo');
// ...
});- Recording Tests: Playwright provides the option to record UI tests using their VS Code plugin. All you need to do is press on the
Record newbutton and Playwright will open a browser for you. Then enter the URL of the web-app and start clicking. The best part about Playwright recording is that it will pick thedata-testidselector (await page.getByTestId('dialog-title');) if one exists.
There are many other advantages to Playwright that I did not cover in this article, mostly because I have not used those aspects heavily, but here are some examples:
- iFrames: Playwright works with iFrames out-of-the-box and the experience is incredibly smooth. We are using iFrames in my current job but not in my previous job (when I was working with Cypress). It is my understanding, however, that you need to install a Cypress plugin to be able to test iFrames. This article explains in great detail why Cypress struggles with iFrames and how to test them.
- Web-app server: Both Cypress and Playwright require your app to be running for tests to be able to run. A popular request for Cypress was for the framework to be able to start the system under test before running the tests. I personally always have my app running when I run my tests, both locally and in the CI/CD pipeline. However, developers wishes have been heard (by Playwright):
// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
// Run your local dev server before starting the tests.
webServer: {
command: 'npm run start',
url: 'http://127.0.0.1:3000',
reuseExistingServer: !process.env.CI,
},
});- Forbid
onlyKeyword: A common use-case, is that when a developer wants to run a single test, they mark it asonly. Unfortunately, they often forget to remove it before committing. Playwright comes with a configuration to detect this, while with Cypress we had to add eslint rules.
// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
/* Fail the build on CI if you accidentally left test.only in the source code. */
forbidOnly: !!process.env.CI,
});
// example.spec.ts
test.describe('Playwright test with async/await', () => {
test.only('should test the async methods used in previous example', async ({
page,
}) => {
await asyncFunction1();
await asyncFunction2();
});
// more tests...- Multiple Pages: Cypress runs your web-app inside of its own web-app, this imposes the restriction that Cypress can only run tests on one page at the time. Playwright on the other hand, can open multiple
BrowserContextand/or multiplePagein each browser. all having different websites/domains/etc. and run a test across them all. - Multi-language Support: In Cypress, you have the option to write tests using either JavaScript or TypeScript. However, in Playwright, the language options for writing tests are more diverse. You can choose to write tests in JavaScript, TypeScript, Java, Python, or .NET, providing developers with greater flexibility in their preferred programming language for test automation.
Conclusion
After having been a fan of Cypress for many years, and a strong advocate of UI testing, it was not an easy decision to start using a new framework that I was not sure would be as good as Cypress is. Having said that, it was such a pleasant surprise to discover that Playwright not only does what Cypress can do but exceeds it in all angles that I can think of.
Lastly, the developer community for both Cypress and Playwright are considerably large. Albeit Cypress still being much more popular, the Playwright community is growing fast as more developers discover how amazing the developer experience is.
I would recommend to anyone who is already using Cypress or is considering getting into UI testing, to consider Playwright as a worthy option.
