JavaScript Web Test Automation Framework Rankings in JavaScript Rising Stars

Early this morning, one mentee sent me this message.

A link to test automation framework ranking on JavaScript Rising Stars.

Compared to 2022 (last year), Cypress dropped from the 3rd to the 5th, with exactly half of the stars.

Before Playwright (the first version was released in February 2020), Puppeteer was the leader.

In 2020 (the same year Playwright was available), Playwright took the crown in Rising Stars, and kept it ever since.

What do we learn from the above data?

1. JS testers keep “trying and dumping” web test automation frameworks.

As a test automation coach, I witnessed and rescued test automation attempts; the failure rate of the above JS automation frameworks is 100%. (See Definition of End-to-End Test Automation Success, and my boosted article, A Tale of a Deceptive End-to-End Test Automation Engineer)

2. Playwright took the JS automation frameworks by storm.

Playwright has topped the JS test framework rising star chart since it became available in 2020. Looking at it from a different perspective sheds light on the shortcomings of those (and many of them) JS automation frameworks.

A real End-to-end Test Automation Engineer just needs to use one Web Test Automation Framework (regardless of language) to test all websites in the last 13 years.

As a test automation engineer (since 2005), I only changed the automation framework once: Watir => Selenium WebDriver (available in 2011).

3. The highly-hyped Cypress is going to disappear

Despite an extensive marketing campaign, including the ‘successful’ Cypress Ambassador program, which is becoming an embarrassment badge, Cypress is on its way to disappearance.

Check out my article, “Cypress.io is dying”.

Selenium WebDriver has always been the correct choice.

Two recent independent surveys showed a similar result: Selenium WebDriver is the framework of choice for web test automation, by a huge margin.

What do I think of the Playwright?

Before sharing my thoughts, you should know that I added support for Playwright to my tools TestWise IDE and BuildWise CT server.

Set up and Develop Playwright Test Scripts with TestWise IDE

Why I didn’t do that for PhantomJS, ProtractorJS, TestCafe and Cypress? Because upon brief examination, those were bad and had no future. I don’t want to waste time. Having said that, I think Playwright is still falling behind raw Selenium WebDriver in most aspects, such as

1. Easy to learn

Check out my article, “Why Selenium WebDriver is So Easy to Learn?”

2. Reliable

You can’t beat a W3C standard-based.

3. Syntax (intuitiveness)

Check out my article, “Playwright vs Selenium WebDriver Syntax Comparison by Example”.

4. Support

My experience with WebDriver support is top-class.

5. Scripting language.

Playwright, compared to other JS-initiative frameworks, added support for Python, Java, and C#. But it missed the most important one: Ruby.

There are some Playwright features I do NOT like, such as ‘auto-waiting’, ‘test runner’, ‘reporting’, ‘parallel execution’, .., etc. I will write separate articles on these topics.

Still, I think Playwright is promising if you had to use JavaScript or TypeScript; I hope it could unite the JS tester community. Over the past six years, fake automated testers using various JS automation frameworks (notably, bad Cypress) have severely damaged the reputation of Test Automation Engineers and Web Test Automation. Now, there are fewer job ads for E2E automated testers, and with less pay and less respect.

As mentioned before, we don’t need that many automation frameworks for testing web apps. Just make it work.

35-Word Functional Test Automation Strategy

Write 1+ Automated End-to-End Test for a user story;

Add it to the regression suite;

Run the regression suite daily and keep them valid!

Starting on the first day and then every day!

----

This original article was posted on Medium (2024-01-22)

My eBooks:

• Practical Web Test Automation with Selenium WebDriver
• Practical Continuous Testing: make Agile/DevOps real