3 Questions you Need to Ask Before Starting an End-to-End Test Project
I like to think that a good coder is like a good gardener. He knows how to organize things to have the least maintenance to do. He also knows how to prepare the ground to ensure the success of his project.
You must care the same when you start an end-to-end testing project, or you can quickly fall into traps that can put you off testing forever. E2E is a great tool but it can also turns into a great pain. To avoid this, you have to ask yourself the right questions from the beginning, even before coding anything. So, I’ve put together some questions to ask in order to best approach an end-to-end testing project.
Is it the right time ?
Since end-to-end testing is one of the most difficult tests to implement and maintain, you should think twice before jumping in. Of course, it is a good mean of ensuring that an application works perfectly from an end-user’s point of view. When properly implemented, it can detect regressions in critical areas while saving time on redundant manual testing.
But before throwing yourself into this beautiful project, you have to ask yourself if it is not too early, or even inappropriate for the product to be tested.
Has the product passed the MVP phase and is the product vision stable enough to build long-term E2E tests?
If the application or feature under test has a possible lifespan of a few months, it is better to find cheaper ways to test it. If this is the case, you should already make sure that you have the necessary coverage in unit and integration tests.
If after that you still have doubts about starting, doubts about the framework you will use, why not make a POC with the most useful scenario to cover ? This will reveal future difficulties for setting up a larger project and provide you good requirements to start without bad surprises.
Is the team ready?
Now that we know that E2E tests are legit, we have to make sure that the team is ready to build them. But who will make it? And when they break, who will fix them? By deciding together now, you’ll have a better chance of empowering the team and motivating them. Whether it’s a dedicated team of testers, developers or a mix of the two, the success of the project will largely depend on its adoption by the team.
Have already a good test base
It is important to ensure that the team already has good practices with unit and integration testing, that they know if a test is useful or not. Simple writing rules allow to reach a consistent ratio of tests based on the customer flow and the test pyramid.
Do not rely on test coverage to find E2E test scenarios. Coverage can help you improve checks in your scenario but not the scenarios themselves. We don’t want to cover the frontend logic 100%, we want to cover critical user flows. For that, I like to use mind-maps and other diagrams to illustrate the flows and see what I cover.
Establish test writing rules
If developers do not have established rules they may fall into 2 imbalances that will be fatal for the upcoming E2E testing project.
If they already don’t code enough tests: Then they won’t have the reflex to code E2E tests either, few good habits, in short, they won’t be ready.
If they code too many tests: Overdosing the testing effort is a good way to demotivate a team on testing. If they already spend too much time maintaining their test base, then E2E testing will finish them off.
Have experience writing BDD scripts
What is convenient with BDD scenarios is that it naturally helps to write some end-to-end tests. By using Cucumber, present on most end-to-end frameworks, you get clear tests, understandable by the product teams and the technical teams. This is a real asset not to be neglected.
Another advantage is that if you have already started designing the product and written down lots of BDD scenarios attached to the features of your application, you will have less difficulty when it comes to cover your application with end-to-end tests.
Gather metrics
How can you measure how much your E2E tests will help your project ?
Of course, there are benefits that you can’t measure, like the confidence it gives you to deploy your app covered with E2E tests. But if you want to demonstrate the utility of E2E tests you need to have metrics before and after the use of E2E tests so you can demonstrate their power.
For example, if you had a process to manual test the critical flows of your application before a deployment, then you should know how long it takes to do it. Once run, your E2E tests should normally reduce this testing time.
An other metric can be the number of bugs and how you found them. Tagging your issues with labels like “foundBy:manual”, “foundBy:e2e”, “foundBy:integration” can help you to track if E2E are useful or not compared to other tests.
Is the development environment ready?
How testable is your application ?
The more you develop your project in order to make it testable, the more it will be easy for you to automatize tests on it. Make it easy to generate data tests — maybe you did it already for integration testing , reuse — , make it easy to authenticate, etc… All these tasks will help you to write tests faster.
How reliable is your test environment ?
Your E2E tests will need a proper environment to be run. Often, the most suitable environment is the preprod, or staging/test. This environment needs to have the same capacity of the production so it will be stable enough for the tests. So it’s better to take the time to work on this environment to make sure it is stable enough before building E2E tests on them.
Other thoughts coming : E2E tests tend to be integrated into a continuous integration pipeline. Does this pipeline exist and is it ready for the E2E testing step?
What’s next ?
Meaning all the requirements have been meet. You made a POC and your team welcomed it with enthusiasm. Answering all these questions has allowed you to set your project on the road to success. Now all you have to do is start and appreciate the journey. Of course, even with the best preparation you will still meet pitfalls. Take a look at best practices and never stop learning how to improve things. But keep in mind that, like every technologies, E2E tests must serve you and not enslave you.
