Playwright Test and Browser Automation
If you’ve been following me for a while, you know that I have a thing for browser and test automation. Previously, I’ve spent quite a bit of time with both LeapWork and Taiko+Gauge from ThoughtWorks.
Despite our best efforts with unit tests, test-driven design (TDD), behavior-driven design (BDD), quality by design, etc. — we still spend an inordinate amount of time and effort on manual testing and even with that effort and cost, teams and products end up with costly defects that can sometimes simply affect usability, other times betray the trust of our users, and in the worst case cause some loss for our users.
Part of this is that developers are really bad at predicting how end users will interact with the software — we tend to be more rational people and think that if only end users would be as rational as we are, there wouldn’t be any bugs! While practices like unit testing are sound in principle, the problem is that developers simply do not think and behave like end users (developers tend to overestimate how rational the end users are). Developers are far too prone to focus on “happy path” testing. Don’t get me wrong, unit tests have a purpose, but that purpose is a really small slice of software quality and testing. The defects my teams have encountered are rarely manifest at the unit level and almost always at the process/workflow level.
Part of it is that the tooling is still a bit raw and the threshold for effort and speed is high. The sweet spot is tooling that is fast and easy to adopt as well as adapt. Ideally, with good tooling, QA and testers can focus on what is known as exploratory testing rather than regression testing. Good tooling supports this shift by allowing low cost, low effort translation from findings during exploratory testing to codified regression tests. If a team cannot efficiently translate the findings in manual testing into automated regression tests, then cycles become longer if quality is to be maintained since the manual regression tests simply accumulate.
To solve for this, we need tooling that meets several criteria:
- Fast. If it takes too long to verify the results, it means the feedback loop will be delayed and that delay creates friction.
- Predictable. If the process is flaky and requires a lot of human analysis, then testers will regress to manual testing as the results cannot be trusted.
- Easy to use and adopt. If the complexity is too high, developers will avoid doing it or will not do it thoroughly.
- Low effort, high return. If the effort is too high, project owners will prefer manual testing since there is really no escaping manual testing of some capacity. In other words, if the effort involved impacts dev time and deliverables, project owners may choose to instead spend their budgets on QA testers instead of dev time.
To that end, there are no shortages of ways that teams have approached this challenge at scale. From Selenium to WebDriver to Cypress and a bevy of other tools, libraries, and frameworks!
How is one to choose?
Today, we’re looking at an open source library and toolset from Microsoft called Playwright which in my opinion, is really leading the pack on many key facets of solving challenges that teams face with modern web application test automation.
In the JavaScript world, it is still relatively unknown. The 2020 State of JavaScript survey reveals that it is still quite nascent in its adoption (at least among the JavaScript crowd) with Cypress being far and away the more widely adopted “default” solution for front end testing.
Playwright is, in fact, a spiritual successor of Puppeteer as the two main contributors to the Playwright project were hired from the team that built Puppeteer.
So should you switch from Cypress? Is it better than Taiko? Let’s find out.
Playwright Compared
The Gauge team at ThoughtWorks actually has a great blog post comparing the different automation tools. This handy chart actually sums it up quite well:
Puppeteer, the predecessor of Playwright in some ways, definitely stands out for speed and framework integration but scores quite poorly on several other fronts.
The good news is that the Playwright team has addressed many of these gaps since ThoughtWorks published this blog post.
Cross-browser support? Easily 4 ⭐⭐⭐⭐ in Playwright as it can automate Chromium, Firefox, and WebKit as well as experimental support for Android and Electron.
Reliability of wait mechanisms? An emphatic 4 ⭐⭐⭐⭐ (watch the video). In fact, the beauty of Playwright and Taiko is that you almost never have to explicitly wait. What’s even better with Playwright is that when you do want to wait, it provides mechanisms for waiting on specific network responses. This solves for one of the biggest challenges with respect to using such end-to-end test automation frameworks which is handling of variation in response times in different environments and under different conditions. With page.waitForResponse, it’s possible for your code to wait for a specific REST or GraphQL response instead of just blindly waiting for 5000ms.
Ease of test failure analysis? 4 ⭐⭐⭐⭐. The Playwright trace viewer is fantastic and provides a recording of not only the UI, but also the network and console during the test run. Each “frame” shows the before and after state of the UI for each step. It makes it incredibly easy to walk through failed test runs. Additionally, there are a variety of options for debugging of tests.
Number of languages to author tests? 4 ⭐⭐⭐⭐. Playwright supports authoring in JS/TS, Python, Java, and .NET! Because Playwright itself is fundamentally an automation library, it can be used with any test runner including Mocha, Jest, NUnit, and more!
Benjamin Gruenbaum’s writeup and handy comparison tool is a more recent evaluation of the tooling and includes Playwright in his comparisons.
Other Key Features
Both blog posts above highlight some of the reasons why Playwright should be under serious consideration by any team doing front-end automation, but there are many, many more including:
- Visual comparisons which compare the visual state of the user interface. This is handy for working with rendered images, charts, graphs, or diagrams or when you need to test for very specific visual layouts.
- Integrated API testing which allows you to use one toolset for your end-to-end UI tests as well as API tests.
- Support for Chrome extensions which can be loaded into the Chromium instance.
- Multi-window, multi-tab support which allows for testing of complex interactions and scenarios including real-time, mutli-user use cases (e.g. chat).
Because core Playwright is simply an automation library, it can even be used with Gauge to provide a declarative BDD style approach to building a test suite when you want documentation/specifications as well as automation.
Closing Thoughts
Where I think Playwright comes up a bit short — at least compared to Taiko — is that it relies heavily on strings in working with the selectors. This has the downside of being a bit more error prone in the authoring phase compared to Taiko’s approach of using functions. The other aspect that I miss from Taiko is the REPL mode which is really handy, in my opinion. But in all other respects, I am a convert; Playwright’s all-around capabilities easily trump these minor gaps.
As mentioned, Playwright itself is not strictly a test framework, it includes tooling for test automation. It is at its core, a multi-platform, cross-browser browser automation tool. As such, there are a variety of ways that it can be leveraged to do interesting things such as monitoring, screen capturing, generalized UI automation, and so on.
If our goal as software engineers, product owners, and QA teams is to produce better, higher quality software faster and with lower effort, then we also need to pick the right tools that support this mission. Playwright is one such tool that gets a lot of it right and the team continues to listen to the community and evolve the toolset.
For any team considering end-to-end test automation, do not miss out on evaluating Playwright.
