Why you should not use cypress for API tests.
The hype is real.
The “new kid” on the block came making bold claims, hiring ambassadors with disruptive talks, and showing that new approaches are as good as the established ones.
No doubt that Cypress is a good tool for browser testing, not so good if you are testing end2end with webpages with iframes or popups, but with a few tweaks here and there, you can make it work.
Angie Jones did a fantastic live “battle” between selenium4 and Cypress (https://applitools.com/event/selenium-vs-cypress-let-the-code-speak/), but I’m here to talk about a particular Cypress function that people are basing their whole automation suite on top of it.
The cy.request function.
Before we move any further, let's understand why we do API tests.
Fast feedback.
I don’t know your current setup to make other claims, but I know for sure we can agree that navigating through several pages to validate a business logic takes time. Precious time.
Also, I assume you and your team want as much coverage as possible as fast as possible to make sure that sensitive flow didn’t break and you are ready to deploy to production.
Sure, you probably have some browser / UI tests with proper coverage, But if you are on the field as long as I do (14 years btw), you know that developers and product owners tend to be impatient.
So, how long an execution time is too much? I don’t know, but as I said, in my experience, people start to be anxious around 5min of execution, despite the coverage and quantity of tests that you have.
Leave it running, and let’s deploy to production because it was just a small change
or
Did it fail? It looks like it’s a flaky test. Rerun it while we deploy.
This brings me some memories, but not the good ones :)
Local execution vs. Continous Integration
Now that we agreed that speed and fast feedback are vital, you also must understand that executing something local is very different than running something remotely on a CI/CD process.
Again, if you work on a startup or small team, the budget for this kind of service or EC2 instance is usually tight, so you have to make the most with less memory and CPU.
Now tell — have you ever run out of memory while using chrome? All those open tabs, just hanging there and suddenly your machine starts to freeze for “no reason”? — Then I have some news for you.
Browsers use CPU and memory.
Multiple browsers use more CPU and more memory.
Suppose you ever used selenium in your life and tried to run parallel tests. You know that you need a selenium grid server (for multiple VM instances), OR you can run it with multiple browser instances opened simultaneously.
This last one makes your machine useless until the end of the execution, and it takes time, like, a lot.
If it takes a lot of resources and time to run browser tests locally, it can take more time if you don’t have enough cloud resources, but even if you do, your performance will most likely be the same as running locally. Why? Because you are using a Browser.
But dude, there are headless browsers that take fewer resources and are faster.
Yeah, but it's still a browser.
It still needs to render js, and it still needs additional resources than a simple tool based on HTTP requests.
The cy. request dilemma
Cypress can run HTTP requests under the hood. It’s like a one-stop shop for all your test needs, and it’s excellent, especially if you don’t have much coding experience.
This method can help your browser tests be faster and more direct, without going around to fetch data. You can make an HTTP request in the middle of your browser test execution with no setup since Cypress re-uses cookies and your current session to authenticate requests.
But the thing is, this is a support method. It’s there to support your browser tests and not the be the BASE of your whole suite.
Why?
Because it’s Cypress, Cypress needs a browser to run their test runner.
If Cypress detects that you didn’t open any page, it will automatically run your tests on headless mode BUT still a browser. It still uses additional CPU and memory to make HTTP requests.
Show me code!
I knew you might say that :)
I put together a simple repo with two cypress tests and two jest +node-fetch specs => https://github.com/leoGalani/api_test_benchmark
You can see the total execution for yourself (if you use Linux or macOS) using time, and I guess you can do the same for windows using measure-command .
jest +node-fetch takes 4s in total:
Cypress using cy.request takes 24s in total:
I know this is not a “solid benchmark”, and my idea is not to do that because someone already did :) => https://blog.checklyhq.com/cypress-vs-selenium-vs-playwright-vs-puppeteer-speed-comparison/
My goal is to make you aware that technical choices have consequences, and at the end of the day, engineers must deliver efficient solutions.
I’m not saying that you should not use Cypress at all, but that if you do have an API suite, you might want to have a separate runner for better performance.
But it’s up to you. If you think it’s better to use Cypress DSL for all your tests or that having only one runner is more important than using resources more efficiently, I understand.
Thanks for reading, and I wish you a great day!
By the way, to make this example possible, I used a service called serverest.dev which is an application that can be used to train API testing and is maintained by the brilliant Engineer Paulo Gonçalves.
