“One tool to rule them all…”
Let’s put an end to the endless fight about QA tooling
For every article, blog post, youtube video about a tool there is an opposite one about why the same tool is not worth implementing, and it’s just because you don’t know how to use it properly. And then if you get to the comments section, there are everyone fighting. It creates clicks, it creates views… and yet who does it really serve ?
The amount of “Selenium is better than Cypress” or “Cypress does it better” or “Did you know Playwright could do this easily?” I’ve seen in the last month alone is astounding. And this is not limited to testing tools, it’s also true for anything that touches to the delivery, dev tools, IDE, monitoring… it’s endless. Maybe it’s because I’m in the Quality field for over a decade now and I have an interest in trying new things, I always stumble on a LinkedIn post, or even just in a discussion at work, where we also have a custom testing framework that tries to cover everything you can imagine.
When you step back and think about it, to try an appeal to everyone you’ll end up managing more and more exceptions as the days go by… and in the end, your initial wish to offer a simple accessible solution ends up being used by a few, select persons. If we step back to testing and automation tools, what is the main goal of your team ?
I’ve heard everything. Selenium is the best tool because it’s open sourced and the community is large. A few years ago we had major issues with our version of Selenium because we used the javascript port. We wanted the devs to contribute to the tests and didn’t want to impose on them that we use the Java version of Selenium. Documentation was not very available and many of the solutions to our problems were gatekeeped by the people maintaining the Javascript repo. We switched to Cypress. It wasn’t perfect but the devs quickly onboarded and we reduced our execution time by half in our pipelines. We tried to execute tests in parallel with Cypress but we hit a wall where we needed to use the paid version to access this feature. Turns out, Cypress is also open source and we coded our own parallel systems with the help of our devOps team.
Our backend team continued to use Selenium, for their tests it was clearly better and in Java. It was filling all the checkboxes we needed, why change a solution that works ? Functional testing in backend was working perfectly, rarely gave us false positives, and almost never flaky. We had issues with contract testing, we needed a reliable robust solution to handle blocking the pipelines in case the schema was not respected by the client needs. We implemented Pact. It was clearly serving it’s one purpose but it served it well. For the node APIs it was not a solution that was easy to implement… we had issues so we decided to stick to Postman and a Newman integration in the pipelines. Our QA analysts happened to learn how to code small Postman tests in Javascript, our backs were covered and it helped the teams gather some autonomy.
Nowadays when I see that there is only ONE tool, I always know this will be a one way discussion. And when I see that there is a solution that merges many solutions I always wonder how this is not a maintenance nightmare…
So next time you see an article about how “this tool is bad” or “this tool is better” ask yourself, what do you gain from putting in place that solution. Your Selenium 3 stack contains over a thousand tests ? Maybe it’s not the time to switch to Cypress… you could probably switch to Selenium 4 and keep a solution that your team is already accustomed to and get all the benefits of the new version. There is no real answer to what tool is the best. The only real question is “What tool answers my need in the best way possible, in that specific context of my project/team?”
