Automatic tests — Are they worth your team’s time?
To automate or not to automate? That’s the question. I have tried to condense in a list of pros and cons if automating your testing will be worth your time or not.
We have been living in a world where more and more things are automated and so we have started to take them for granted. However, sometimes it gets forgotten that something automatic requires time to be made and set up in order to work. This is specially tricky in the “Testing World”.
Often we hear of teams that have been asked to automate as much testing as possible. Also that the business is really looking forward to a test suite able to automatically validate their new and shinny systems. However, often it proves difficult to explain to “non-tech” people what are the implications on automating all the testing and the impacts on the team and their velocity. This is why I have tried to condense all the pros and cons of automating each level of testing (according to the testing pyramid) so it can be a bit easier to make the decision of what should be automated (or not) for a specific project.
Depending on the project we will need to decide what is the right approach to testing. Some projects will require to meet high-quality standards in order to be delivered (i.e. high profile app for a bank) while others may only require basic validation in order to be released (quick Prove Of Concept that will only reach a small subset of users)
Test Levels
Level 0 — UNIT TESTS
Unit tests have two main functions. They make the developers think on how their code could break which leads them to write cleaner code. Also, they provide confidence in the moment of changing the existing code. Unit tests might seem really expensive at first sight. However, they really pay back the longer your project lives. So, automating them should be consider (almost a MUST have) for every piece of code that will leave longer than a Prove Of Concept.
Level 1 — INTEGRATION TESTS
Integration tests are responsible to validate that 2 (or more systems) are able to communicate properly between them. It is very common that 2 “perfectly” working systems break because the integration between them is not right. It is for that reason why they should be automated and provide feedback as soon as possible in the SDLC. However the approach to Integration Testing should be very pragmatic and make sure that they never test functionality but only “integration”. That will help reducing the cost and improve their efficiency.
Level 2 — END TO END TESTING
End to end testing is usually a very controversial area to automate so I have decided to divide it into 3 different sections so it can be easier to identify where to put your efforts (and money).
Level 2.1 — FUNCTIONALITY TESTING
Functionality tests are usually expensive in time to create and run. That’s usually why they tent to be manually executed. At this stage there isn’t any rule on when to automate or not. It really depends on several factors as QA Team size and skills, change frequency expected around the functionality, etc. Automate functionality testing should be carefully studied and not taken for granted.
Level 2.2 — EXPLORATORY TESTING
The table speaks for itself and exploratory testing proves to be really difficult to automate with current tools. The return on it does not tend to be extremely high either.
Level 2.3 — SMOKE/REGRESSION TESTING
The idea of testing is to provide feedback as soon as possible to reduce the cost of fixing an issue. Regression testing are the easiest place for a “non-tech” person to understand what that exactly means. Automating this test scenarios will provide really quick feedback to a developer on a time frame that it will prove impossible for a manual tester (specially in big projects). On the other side they have the same inconveniences to automate as the Functionality Tests. Again, the decision on automating these tests should be made considering almost the same bullet points as the functionality tests.
NOTE: All “saving time” benefits are related to the first round of development. Time will be impacted significantly once the development team start facing issues.
Summary
In order to achieve high quality standards, automation testing is a MUST. Why do I say that?
Even if in general, the development speed will be increased when working on new functionality. That increase in velocity is not just due to less work, but also to overlooking requirements or defects. If the overlooked requirements or defects are captured during a manual test round which happens right after the development process has finished, then extra work will be required from the development team and consequently reduce their capacity to deliver new functionality.
In the case that those missed details make all the way to production, they will have even a higher impact in the development cycle. Not only due to complexity to fix issues but also on customer satisfaction and business value.
So, even though the costs of your project might seem higher than expected at its very beginning, the return on the investment will be clear once the project reaches maturity. Anyway, always approach the testing pragmatically and adapt it to the needs of your project.
As Martin Fowler mention in his article about the DesignStaminaHypothesis, good design means higher velocity in the future. Same happens when the right level of testing is achieved.