Quality Assurance and Software Delivery Processes in Frontend Engineering pt.2
How we improved SDLC of web apps at Azimo
In the previous blog post, we described our key achievements in introducing Quality Assurance and improving the software development process for our web applications. In this article, we will look closely at the key projects which helped us to achieve our goals:
- On-demand continuous delivery,
- QA culture — responsibility for quality distributed across the entire frontend engineering team,
- The faster delivery process and even higher quality of our product.
Here are some key initiatives which helped us with this transformation.
Code coverage gateway
For a long time, unit tests were just nice-to-haves in our web apps. Engineers wrote them, but we didn’t have precise goals and standards. After we decided to introduce Quality Assurance to the entire engineering team, within a few months, we increased our code coverage from 15% to 50% and later to >80%.
We covered some reasoning behind unit testing in one of our previous articles.
In frontend engineering, unit tests helped us with:
- Writing testable and well-architectured code,
- Saving QA engineers time spent on manual testing,
- Increasing speed of the release cycle,
- Building a base of the testing pyramid.
We introduced a code coverage gate to achieve our goals and move responsibility for quality to software engineers. The automation cross-checked each Merge Request with the main branch for the amount of code covered by unit tests. And then, a CI job passed only if code coverage was higher or equal. This approach ensured that we covered with tests all of the new features and encouraged engineers to add some tests to existing code.
From Protractor to Cypress
We built our initial end-to-end testing stack on top of Protractor. While now we see that this tool wasn’t bad, it wasn’t the right one for what we wanted either. Our tests took a long time to run, and in worst cases, they reached up to 70% of flakiness. Errors were lengthy terminal messages with not much helpful information. It all made testing and fixing bugs quite difficult and time-consuming for us.
Over time, we migrated the testing stack to Cypress, which appeared to be the best choice for our needs. This framework runs tests in an interactive browser environment in real-time, imitating the real user behavior.It allows us to automatically wait for commands and do assertions before moving to the next step. We can use Developer Tools directly to figure out the cause of an issue. Cypress also has great error messaging, extensive documentation, and a dashboard for tracking test failures, flakiness, and screenshots of failed tests.
Cross-browser testing
One quick add-up to our End-to-End test suite was introducing cross-browser testing to ensure that our website delivers an optimal user experience, independent from the user’s browser of choice. With Cypress, we can run tests across multiple Chrome-family browsers (including Electron and Chromium-based Microsoft Edge) and Firefox.
When running Cypress tests in Test Runner, a complete list of available browsers is displayed within the browser selection:
It can also be specified using the — browser flag when using the run command to launch Cypress, as well as in an npm script as a shortcut:
Automated email testing
Email testing was an area long neglected and tested only manually before each release. It was time-consuming and not reliable enough to pinpoint issues in critical workflows. After some research, we have settled on Mailosaur, because of its easy set-up with Cypress.
Mailosaur uses a new email address for every test run (unless specified otherwise) and enables us to check real-world test scenarios that perform common user actions, such as following links in emails.
As appealing as the functionality was to us, we were using redirects in our email links, and Cypress only allows one superdomain per test. We had chromeWebSecurity set to false in the cypress.json file already, but our tests had to click the link and visit Azimo using the redirected link to replicate the users’ behavior truly.
After some research, the best workaround was forcing a visit on an email link using a custom command:
Now the tests are following the exact journey Azimo users would take. We can verify scenarios like account verification, password reset, or opening transaction details directly from the email.
Visual regression testing
Regression testing ensures that any changes we’ve introduced to our code do not have an unexpected impact on our application. Visual regression tests do precisely that but focus on the interface instead of functionality. Initially, changes done to our UI have been checked manually, but it was time-consuming and error-prone, as we have multiple browsers, mobile devices, and screen sizes to cover.
To help with this issue, we have settled on Percy to cover our single-page application. Percy is famous for being easy to use, flexible, and, importantly for us, offering a way to integrate with Cypress. Thanks to that, we could reuse our test scenario steps to create 2 End-to-End tests — one for the individual users and one for business, covering visual changes for all the most critical parts.
Percy comes with an intuitive dashboard where changes can be reviewed and approved/declined. By default, Percy uses the latest master build snapshots as a baseline for comparison. It generates visual diffs to see how our changes will look on mobile and desktop and four browsers — Safari, Firefox, Chrome, and Microsoft Edge.
To take Percy snapshots, we need to use the cy.percySnapshot() command. But to make it simpler and allow us to reuse it for all the snapshots, we came up with the following command where the file name is defined in the test scenario:
In an End-to-End test scenario, it would look like this:
Visual regression testing is an excellent way for us to validate regressions and ensure that no small change in the code will result in a broken UI.
Reducing test flakiness
While having a comprehensive testing suite is great, it is arguably even more important to be able to trust it, meaning reducing the flakiness.
A flaky test is a test that will pass sometimes and fail other times, even though nothing has changed in the code between test runs. Due to this unpredictability, it makes it difficult to fix it. A couple of changes that helped us tackle it was adding test retries and reducing inconsistent interactions in the DOM.
Step 1 — tests retries
To know if the test is flaky or failed due to environmental instability (e.g., temporary outages, API response delay), we often had to rerun them multiple times, which is time-consuming. Lucky for us, Cypress has introduced test retries functionality. It allows us to retry failed tests to help reduce flakiness and continuous integration failures while saving valuable team time.
Test retries aren’t enabled by default in Cypress. We need to configure that in the cypress.json file to have an X number of retry attempts.
It is possible to define retries for individual tests and test suits, however, we decided to configure it globally, with the following options in the cypress.json file:
- runMode allows you to define the number of test retries when running cypress run
- openMode allows you to define the number of test retries when running cypress open
In the case above, Cypress will retry the test 2 times (3 total attempts, one when the test case was run the first time and 2 more attempts of retry) before it will be marked as failed. If the test passes the first time, it will move on to other test cases.
We found Cypress retries beneficial, especially during working hour runs when our staging environment is experiencing more traffic.
Step 2 — Reducing inconsistent interactions in the DOM
One issue that kept reappearing in our tests was elements in the DOM that did not render correctly, causing flakiness, especially in our CI pipeline. When checking manually, elements looked fine but were producing timeout errors when running End-to-End tests.
Timed out retrying: Expected to find element: [element], but never found it
Timed out retrying: cy.click() failed because this element is detached from the DOM.
To understand how to fix it, we had to look closely at how Cypress retries commands and assertions. By default, Cypress will retry commands .get() and .find() while it re-queries the DOM until it either finds the element or times out. However, if you have a chain of commands with an assertion such as:
Cypress will retry only the .get() command which is prior to the assertion which caused the failure, not retry the entire command.
While it is a viable option to use the cy.wait() command, it needs to be used with caution, as it can significantly impact the speed of the tests and, more importantly, hide actual bugs. If the user with a slower connection would try to interact with the element in the DOM that does not have data loaded yet we could disallow it until the element is completely loaded, but by making Cypress wait, we might not be aware of the issue.
To combat the issue, we combined our .get() and .find() commands into one and then used an assertion in a single command:
Using the new command, we knew that .get() would be retried if the assertion failed.
Changes in the delivery process and transformation from Quality Control to Quality Assurance don’t happen overnight. That’s why we are proud of our achievements and projects which we described in this blog post. I hope these materials will inspire you to make your web apps truly exceptional.
Acknowledgments
All of our achievements described in this series wouldn’t be possible without the hard work of the entire Web engineering team. The biggest credits go to Artur Kania, Software Development Manager. Together with the web team, we built the foundations for the changes described in this series.
Towards financial services available to all
We’re working throughout the company to create faster, cheaper, and more available financial services all over the world, and here are some of the techniques that we’re utilizing. There’s still a long way ahead of us, and if you’d like to be part of that journey, check out our careers page.
