In my software testing career I have heard many phrases like “We don’t need testers in a team!” to “We need to automate everything with 100% coverage!.” Everytime I heard these phrases I was shaking my head. I bet you know why. Both extremes are wrong, because I think every software development team should have a tester on board. And of course 100% test automation coverage is not possible, efficient and required.
In this article I am sharing my lessons learned in test automation that I learned the hard way working for more than 13 years in the software testing industry. I will share a closer look at test automation and the pitfalls to avoid, that I made myself in the course of the last few years. I will share the dos and don’ts I see when it comes to test automation, to help you not make the same mistakes.
Dos in Test Automation
So what are the things you should do when working with test automation? Let’s take a closer look.
Hire the right people
Hiring people with software engineering skills is a must. Without the right skill set, your test automation will fail. Therefore, it’s required to invest lots of time in the recruiting process to identify the right people for your needs. I recommend you to write down your goals that you want to achieve with the new hire. Once the goal is clear, derive concrete skills from it.
Now it’s time to write a catchy, not too long job description with the core skills needed. Don’t overdo it with technical terms and requirements in long bullet point lists. This might have a negative impact on potential hires.
When talking to people in the interview process, technical skills are a must. You can verify them in coding challenges or code reviews that you are doing on the fly. However, don’t just look at the technical skills. It’s also very important to check for team or company fit and the right mindset when it comes to working with people, communication and problem solving.
Once you have hired the right people to work on your automation, trust them to do the right thing. Give them as much context as possible, bring them together with other disciplines in the company such as developers, designers and product people. In best case each development team also has a test engineer on board.
Take your time in finding the right test automation tool
Having the right test automation skills in the company is a big plus. The next challenge is to find and select the right test automation tool. And this takes time. Before selecting a tool sit down with all involved parties to define the goals of the test automation. During that process try to raise as many questions as possible towards the tool, the own environment and the product that will be built. With the help of these questions, you will create a long list of selection criterias that will help you to find the right test automation tool for your needs.
The list can have the following criterias:
- Does the tool support different programming languages?
- Is the tool able to automate against different operating systems?
- Does the tool support an integration into CI/CD pipelines?
- Does the tool offer reporting capabilities?
- Is the tool well documented, offers support or has a great community behind it?
- Does the tool fit into the allocated budget?
- How complex is it to set up the tool?
- Does the tool offer a flexible test execution e.g. on different operating systems or environments?
The above list contains only a few possible questions to raise to find the right tool. Therefore, it is required to invest enough time to find the right tool for your needs and environment. It can also be that you need more than one automation tool for different layers in your tech-architecture, or alternately, working with a test automation management platform additionally to the test automation tool. Then you should run the selection process for each tool separately.
Start simple
Once you have identified the right automation tool for your needs, it’s time for the configuration and setup within your team or company. In the first few days or weeks, try to learn all about the tool and its features. Once you and your team are familiar with it, start working on the first automation scenarios. But always start simple! Take a look at product features that are easy to automate and help the team to focus on other parts of the quality. When this test is robust and delivering reliable results, get to the next level of automation. Start working on more complex scenarios and see the results. It’s also recommended to work on the product parts that will not change on a daily or weekly basis. Usually the core features of a product are the perfect starting point for automation.
Involve developers in the automation process
Writing test automation code isn’t easy. It has the same complexity as writing the production code and this should never be underestimated. Usually a software development team has 3-5 software developers and at least 1 software tester. This ratio is ok. However, the test engineer has a hard time catching up with all the code changes made to the product. S/he needs not only to test the software manually but also needs to write the complex automation. Doing this all together isn’t possible for one person on the team. Therefore, it’s a must that the software developers are also responsible for writing the test automation code. Not only on the unit level, but depending on your automation strategy for example also for the API, the integration or the end-2-end automation layer.
Each team should sit together and should talk about the automation process. It must be clear to everyone that the quality of the product is everyone's responsibility. This mindset requires everyone in the team to support automation. If this is not the case, it is very likely that your automation process will fail.
Invest time in the automation pipeline
Similar to the time investment of the tool selection. A software development team must invest enough time in the set up of the automation pipeline. The pipeline must be configured in a way to support the needs of the development team. For example, to run on every commit, to run on a pull request or to execute the whole test suite during the night.
It’s also important to define which tests on which layer should be automated when. In the best case the team is working on an automation pipeline strategy. Once the pipeline strategy has been implemented, it’s important that the results are made transparent for everyone in the team and even better it’s transparent for everyone in the company for example with a dashboard.
The work and the invested time on the automation pipeline should not be underestimated. Depending on the project or product size, this can be a full time job for a single person or even a whole team.
Don’ts in Test Automation
Now that you have learned about my dos when it comes to test automation, let's take a closer look at the don’ts.
Don’t select a tool just because it’s hyped
My number one point of the don’t do list is, when selecting a new test automation tool, just don’t blindly select a tool just because it’s hyped in the software development or testing community. If you are really lucky, the tool might work for you, but in most cases you will fail with it. As I mentioned in the Dos section in the article, selecting a tool requires time. Without this investment in the selection process you will probably lose more money in the long run, because at some point in the product development process, you notice that the tool is not supporting a special feature or is not able to scale the way you need it.
When selecting a tool, it’s good to check what tools are hyped at the moment and why. Maybe there is a valid reason for this hype and you might benefit from it. It’s always good to keep up with new tools and overall with the communities to stay connected and to get the latest news.
Don’t try to automate everything
I already mentioned this in the intro of the article. Whenever somebody in your team or company tells you to automate everything, just don’t listen. This person is likely someone who never worked in software development or has no idea about test automation. I have seen inexperienced people, who were new to automation who tried to automate everything but they soon noticed, it’s just not possible. When you see people with this view, talk to them and explain the negative side of it.
What you should do instead is to take a look at your product and code architecture and to identify the critical parts of your application. Then take time to create an automation strategy for those parts.
With a clear focus and strategy in place a handful of automation scenarios can make a much bigger impact to the overall quality of the product.
Don’t automate too soon
Also a classic mistake when starting with automation in a product development team, is starting too early with it. If the team is developing a brand new feature or product, it’s really likely that the features will change. In this case it doesn’t make sense to start with the implementation of the automation. The invested time might be wasted. Instead the team should start with the foundations. For example, setting up the CI/CD pipeline, think about the data structures needed for the feature. From that derive a test data strategy and maybe prepare scripts for the test data generation.
Once the features are getting more and more mature, start slowly with the automation. I recommend having a close relationship with the product owner, to understand the upcoming product features and changes. Use this information to create your automation strategy.
Never replace manual testing with automation
And last but not least, the “don’t” you should never do, is to replace manual testing with automation! Manual testing is such a powerful part in the development process of a product, it should never be replaced by a machine. Why? Well, the automated tests will be executed by a machine/ computer. The machine is doing exactly what has been written in the test automation code and nothing more. This is not bad and we all know that this is useful information, but a software tester will find many more issues while performing manual testing. First of all the tester is acting like a user. Using the product a user will use it, for example with a mouse, keyboard or with the fingers using all his/her senses while using the product. In most cases, manual testing will find issues that nobody ever expected to happen.
Therefore, test automation is an important addition to manual testing.
Summary
As you have seen in this article, one can make many mistakes when working with test automation. There are many topics to keep in mind when hiring automation experts, finding the right tool and finding the right moment to start with automation.
If you have a clear goal and focus of what you want to achieve with test automation and you follow the Dos from this article and you leave out the Don’ts, you will be successful with your test automation in your product team or company.
Let me know what you think about the Dos and Don’ts and I would love to hear your stories about test automation in the comments below.
#HappyTesting
