Don’t let Flaky test cases affect your automation results; How to avoid and deal with flaky test cases in Cypress
What is a flaky test case?
If you have noticed few of your test cases have failed intermittently in the same environment in several runs; Congratulations, you have found some flaky tests in your code. Sometimes the tests might be working fine in your local environment, but those tests fail when they run in the CI/CD pipeline. This will directly affect the correctness of the test results.
You can identify these flaky tests by monitoring the test results of your Cypress tests of several runs without doing any code change. Sometimes those test cases fail and sometimes pass.
Flaky test cases are big red flags as there can be reasons for bug leakage.
What are the reasons for Flaky test cases?
There may be many reasons for a flaky test case. Few of them are,
1. Test Data — Sometimes test data such as dates, and random values can be changed in test runs due to data corruption and dependency on other test data. Hard coded data can be a reason for this flakiness.
2. Timeouts — Test steps that depend on loading time can fail due to the required elements/test steps not loading. The main results of this issue are network inconsistency and delays in the network.
3. Environment Dependencies — Differences between the local environment machine and CI such as network speed, number of CPUs, libraries, and the differences in operating systems can cause this. If the environment is a shared one, then it will wave flaky tests.
4. Unpredictable test results — This will be a cause of flaky tests as sometimes test results are unpredictable due to the test case order or parallel execution, this will generate random issues.
5. Bad test design — Because of a poorly designed test framework, bad test case order or lack of error handling can also make flaky test cases.
How to fix flaky test cases?
If you find any flaky test, you need to log it and try to fix it as soon as possible as that test will be a potential bug and you might miss the bug by considering that it’s only a flaky test case.
1. Adding retries
You can add two types of configurations to achieve retry as per your need
a. Global Configuration — This will define the retry attempts for runMode and openMode.
runMode is used when you run your test suite by using cypress run command.
openMode is used when you run your suite by using cypress open command.
These values should be passed in the cypress configurations file.
You can use the bellow code if you want to configure the same retry attempt for both cypress run and cypress open .
b. Custom Configuration — You can configure the number of retries in the test case/test suite level by adding the below code to the test’s configuration.
2. Add exist and visible assertions before clicking on an element
Before performing an action on an element is better to locate the element by using .should(‘exist’), .should(‘be.visible’) assertions.
3. Use cy.wait(time) command
This custom command will wait for the given time to load DOM elements. But this is not always the best way to fix the flakiness as this arbitrary wait will make your code slow. So use this with caution.
4. Reduce inconsistency in requests
When tests make network requests, they can be inconsistent. Tests will continuously run even though the network request does not return a response, and gets fails.
To make sure you get a successful response, you can use the command cy.intercept() .
When to be careful and prevent flaky test cases?
Most of the time flaky tests are expensive to repair. It takes more hours to debug since the actual reason is not visible. So, it’s always better to try to avoid creating flaky test cases when writing the code.
Be careful when you write your test case if you see any red flags mentioned below.
1. If you do not have an isolated test environment for test runs
2. When your tests depend on previous tests
3. All the tests using a fixed waiting time
4. Writing a test script for complex functionality, so the code has many lines and many logics
5. When you use unpredictable test data such as date and time
Conclusion
It’s always easier to write proper tests than to maintain and debug flaky test cases. So, try to avoid having flaky tests in your Cypress test suite. If you already have a few, then try to quarantine and fix those.
Happy test scripting!
References :
1. https://www.cypress.io/blog/2021/05/06/live-webcast-flaky-test-management-with-cypress
2. https://docs.cypress.io/guides/guides/test-retries
3. https://docs.cypress.io/guides/cloud/flaky-test-management
