What automation framework to choose in 2023
A guide to picking the best modern framework for your requirements
As the new year got underway, many businesses conducted their annual assessments and found that writing quality tests is less expensive than releasing four hot fixes in a single release cycle. Thus, testing has once again become a hot topic, and as software automation becomes more and more vital, demand for QA automation experts is rising quickly.
Therefore, it would appear that this is a wonderful moment to begin learning about test automation or a new automation framework. But given that ChatGPT is the topic of so much discussion, let’s see what it has to say.
Let’s start by examining the automation frameworks that will be useful to know in 2023. As there is no such thing as a magic bullet, we will cover different approaches to various situations.
As Javascript is used for front-end development, it makes sense to utilize it for end-to-end testing as well, which is why I’ll be concentrating primarily on Javascript frameworks in this post. Nevertheless, the majority of the contemporary frameworks discussed here support a variety of testing languages.
Cypress
Cypress is a prominent modern framework that has grown in popularity in recent years. It’s frequently a great tool for beginning with automation because it’s quick, dependable, and simple to use.
Let’s look at several benefits of using Cypress.
Simple setup, really easy to use and understand
Cypress is relatively easy to set up, has excellent documentation, and has a fast learning curve. The functionalities the framework offers are relatively simple to comprehend and use, even for those with only a basic understanding of Javascript. Here’s an example of a simple test:
describe('Test clicking on an element', () => {
it('Clicks on an element and checks text', () => {
cy.visit('https://example.com');
cy.get('#element-id').click();
cy.get('#result').should('contain', 'Expected text');
});
});Support for multiple types of testing
Cypress supports API testing, E2E testing, visual and component testing. Combining them all can satisfy the majority, if not all, of a project’s automation requirements.
Modern features
In order to speed up development and debugging, Cypress offers an auto-await function, a built-in test runner window, and the ability to record videos as well as screenshots of tests.
Speed and reliability
The tests run more quickly and reliably because Cypress does not use a driver and communicates directly with the browser.
Great community and popularity
Cypress is used frequently due to its simplicity and cutting-edge features, and according to the most recent State of Javascript survey conducted in the US, its usage is still increasing at a slow but steady rate. The 42k stars on Cypress Github demonstrate how powerful the Cypress community is.
Now lets take a look at some of the downsides of using Cypress
- Does not support testing on Safari (experimental for Webkit)
- Only supports Javascript/Typescript
- Parallel testing is complicated
- Does not support testing in multiple tabs
- Does not support multiple browsers at the same time
- Can’t test native mobile applications
- Using Page Object Model (POM) is discouraged (more on this topic)
- Forbids using concepts like async/await and promises (more on this topic)
More on the permanent limitations here.
So should you choose to use Cypress in 2023?
Well… it depends. Cypress can be a fantastic tool to start with if you are new to the automation world. If you’re not familiar with ideas like promises, the fact that Javascript’s asynchronous nature is largely hidden from the user can be a benefit. If you’re testing a single-page web application, Cypress can be a reliable and simple solution to automate the process. Great community can also be a deal breaker for new users.
However, if the application being tested is more involved and requires numerous tabs to be open or to be tested on Safari, alternative options may be preferable. The following framework might be a better option if you have more experience with Javascript and programming in general.
Playwright
Playwright is a framework developed by Microsoft that offers modern features with relatively little limitations. It provides a great environment, lots of built-in functionality, and integration possibilities.
Let’s look at several benefits of using Playwright.
Support for multiple programming languages
It can be difficult to switch from one automation framework and language to another (often from Selenium in Java), thus Playwright supports many languages, including Python and Java. However, it is advised to use Javascript or Typescript.
Support for multiple browsers
Playwright supports executing tests on all major browser engines like Chromium (Chrome), Firefox (Firefox) and Webkit (Safari).
Speed and reliability
Playwright does not use WebDriver either; instead, it uses the DevTools protocol to interact directly with the browser. This ensures stability and provides much faster headless execution.
Modern features and built-in tools
Playwright supports parallel execution and auto-awaits out of the box. For simpler debugging and test creation, it also offers a suite of tools like Playwright Inspector and Playwright codegen.
Support for multiple types of testing
API, visual, and end-to-end testing are all possible using Playwright. Although the feature is currently in beta, it can also be used for component testing. Lets take a look at a simple API test written in Playwright
test('Should create a product', async ({ request }) => {
const newProduct = await request.post(`/product`, {
data: {name: 'Shoes'}
});
expect(newProduct.ok()).toBeTruthy();
const products = await request.get(`/products`);
expect(products.ok()).toBeTruthy();
expect(await products.json()).toContainEqual(expect.objectContaining({
name: 'Shoes'
}));
});Variety and easy integration
Playwright offers a built-in test runner by default and supports multiple assertion libraries, but it is also easily connected with Mocha or Jasmine.
Great documentation and satisfied users
The Playwright documentation is excellent and has lots of examples. It ranked first (out of end-to-end frameworks) in terms of user satisfaction in the most recent State of Javascript report.
Now lets take a look at some of the downsides of using Playwright
- Can’t test native mobile applications
- Requires basic Javascript / Typescript knowledge to get started
- Fairly new — so the community is somewhat smaller
So should you choose to use Playwright in 2023?
If you are fairly familiar with Javascript and are not currently testing mobile applications — then definitely. It supports all of the main browsers and offers outstanding functionalities and incredible tools. Aside from that, it is supported by one of the most powerful corporations in the world, therefore the likelihood that it won’t be updated frequently is pretty low.
We discussed two modern frameworks for two different needs, yet there is still some confusion. What automation framework should we use if we need to test native mobile apps?
Well the answer is probably a combination of two frameworks — Appium and WebdriverIO
WebDriverIO + Appium
WebDriverIO is a Javascript automation framework that uses Selenium internally and extends its features with some modern functionalities.
Because it leverages WebDriver, testing is available across all browsers. However, because there is an additional layer in the communication with the browser, tests are slower and less stable.
Compared to Cypress and Playwright, WebDriverIO is a little more difficult to get started with, although it is still easier than Selenium.
Despite having several issues because of its connection to Selenium, WebdriverIO can be used to test native mobile applications.
That is usually done with the help of Appium
Appium is an open source test automation framework for use with native, hybrid and mobile web apps. It drives iOS, Android, and Windows apps using the WebDriver protocol.
So should you choose to use Appium + WebDriverIO in 2023?
Well… Yes, Appium is unquestionably required if the project you’re working on is focused on testing native mobile applications. In comparison to Selenium alone, WebDriverIO can offer simpler setup and integration.
Why not Selenium?
As was already discussed, Selenium leverages the WebDriver protocol to add a layer to the communication with the browser. This indicates that the tests are considerably slower, frequently unstable, and difficult to maintain.
Selenium has undergone virtually little modification in the past ten years and does not offer the contemporary tools and capabilities that modern frameworks do. Additionally, utilizing Selenium is very difficult to begin with because it necessitates extensive setup and prior knowledge.
However, customer happiness is the most crucial factor. Users are unhappy utilizing this framework because of its restrictions, as evidenced by the interest statistic of the mentioned survey.
Conclusion
Playwright appears to be the most promising option this year and to be here to stay. However, with more businesses using it, Cypress is still going strong.
One thing is still true. Automation frameworks are merely tools; learning how to select the right tool and determining where to apply it are more crucial. Hopefully, this post assisted you in choosing a tool for your issue.
Cheers. 🍻
