How 3 Process Patterns will change the way you do Test Automation
A few years ago, here at Bumble Inc — the parent company that operates Badoo and Bumble apps — we started to shift towards using autotests more in our development, testing and release processes. We have, however, encountered a few challenges along the way.
Before we start, you might be curious to know why we have been investing in Test Automation in the first place. The answer is simple: our mobile applications have weekly releases. For the QA department, this means we need to test the feature itself, integrate it into the application and do the regression. Effectively, complex regression testing must be performed at least once per week, which is frankly exhausting.
Test Automation was our response to this situation and was intended to make releases faster and more reliable (read the full story in more detail here).
As you know, when trying to resolve an issue in software development we often use a pattern — a general, reusable solution to a commonly occurring problem within any given context. Test Automation is no exception, there is even a handy Test Automation patterns wiki reference. In this article, our main focus will be on Process patterns for Test Automation. As their name suggests, they show how the Test Automation process should be set up or how it can be improved.
Without any further introductions, here are the issues that we’ve struggled with the most and some recommendations on how to overcome them.
Ask for help
Initially, our QA team wasn’t very confident with automation tests. As often happens in engineering, our QA team is diverse: we all have different backgrounds, experiences, and domains of expertise. It’s important to highlight that back then not everybody in the QA team had been exposed to Test Automation. When we decided to invest more in autotests, it was a steep learning curve for some of us. To help our QAs overcome their new challenge, we encouraged them to ask the Test Automation team for help. The reasoning behind this was that if you get stuck on a particular problem, you shouldn’t ponder too long on it — others may have faced the same issues and so perhaps there’s already a solution. Still, in order to avoid overwhelming the Test Automation engineers with loads of requests, we decided to formalise the approach. The ASK FOR HELP Process pattern seemed to be a wonderful route to go — Ask for help instead of wasting time trying to do everything yourself. After all, no single team is going to have all the skills needed at any one time but instead, having a pool of experts to turn to, is the way to go.
As it turned out, merely having a pattern to follow can cause problems. The Test Automation team started getting even more questions than usual (for example, on issues around environment setup). Over time, we often noticed the same questions were being asked by different team members. It was taking significant time on the part of our Test Automation team to respond to all those people because even a simple well-understood issue requires context switching.
However, for most cases, we do have documentation and solutions for common issues on our local Wiki. It may be not the best way to document all possible errors and explanations but we thought it would be useful to have them stored somewhere.
So, we came up with the idea of local StackOverflow. Our current solution is based on the Open Source solution — Scoold. It is designed to act as the first line of inquiry. It works like the common StackOverflow but is limited to the people inside the company. Whenever somebody has a problem, it will be enough to use the local StackOverflow: either to find the answer or to post a new question and get a reply from the expert team.
Potential pitfalls:
- No responsible person to kickstart the implementation. Even the simplest solution requires somebody to drive it through. Luckily, we had such a kind of person. It’s equally important to ensure that management both understands and supports the initiative.
- Low adoption rate because the platform is no longer as popular as it was supposed to be. Any new tool or technology is going to struggle to win the hearts of its potential users — understandably, not everybody likes changes. We might not have been super-active on the website from day one but gradually, we were able to effectively re-route almost all the relevant questions there, as well as bring in people from outside of the Test Automation team as problem solvers.
- Responses might not be prompt enough. In this case, consider implementing notifications to messaging apps about new questions. Especially these days when many of us work from home, and our messengers are buzzing with incoming notifications — it’s more tempting to switch all alerts off. Every team will have their own solutions to this problem, but for us, a reminder in the common Test Automation chat room in Slack seems to be doing a good job.
Benefits:
- Process gamification, so more people are involved in resolving issues. Note, that we didn’t invest heavily in different badges or other perks. In fact, the number of upvotes or stars a person gets doesn’t mean anything. We believe you shouldn’t cross the line here. Gamification is only an additional stimulus for participants, not the main actor.
- Less time and effort needed to resolve common issues. First of all, issues became clearly visible, so it was easier to fix them, instead of suggesting workarounds. Secondly, we reduced the circuit of redirects from one person to another and avoided the same message being forwarded to different people.
- Fewer messages and threads in common chats, where they are hard to track and find. Messengers are arguably the best way to store your docs and to refer to them later as a source of truth. StackOverflow is much more convenient.
Set standards
Whenever possible we seek to share code across the apps/platforms to facilitate maintenance and support. But as with any other system developed over the years by many contributors, different parts can be written in different styles. Our main language for end-to-end autotests is Ruby (to explore the history of Mobile Test Automation in our company click here). In Ruby, intentionally, there are several ways to do the same thing. This is because Ruby’s creator Yukihiro Matsumoto wanted developers to use whichever way was natural to them to get the job done. Furthermore, as you remember, we all had different previous experiences and so might have coded in different ways. Consequently, these all meant it may be challenging to work on someone else’s code.
SET STANDARDS process pattern — Set and follow standards for the automation artefacts — was chosen to deal with the problem. Without that, it might be that many people work on the same project, but all use different styles and methods (imagine they “speak different languages”). For this reason, they are unable to share their work efficiently. We started by gathering feedback from the individuals who would be using the automation framework and tools. In this way, we could ensure that the conventions we adopted would serve all the users. Moreover, anyone involved in the decision-making would be sure to be more supportive of it in the future. You wouldn’t like having some arbitrarily imposed standards if people don’t understand the reasoning behind them. Thus, we’ve worked up the common guidelines that are shared and adhered to by all team members and helped us to find a common language between us all.
What we have done:
- We have set up a local wiki with up-to-date guides and documents (i.e. how to set up the environment, run the tests, and support the tests), which also help shorten code review cycles. We use Confluence for all documentation and have split it into a few spaces. All pages are periodically reviewed and looked after by their respective teams.
- We have added cross-team retrospectives and conventions to the local wiki. Meetings took place on a monthly basis and there were actionable follow-ups in 100% of cases.
- As regards the test code itself, and the restrictions on the code level for the basic verification methods and steps, here we have introduced standard tools, including Rubocop (Ruby static code analyser and code formatter), guard checks and pre-commit hooks.
- We have cleaned up the legacy code and extracted universal and parameterised steps and methods. Everybody is advised against creating their own methods where they can re-use the existing ones. Check out our sample project with some of the best practices here.
- We have set up a code quality committee made up of Test Automation team members. It has a rolling list of questions and tasks everybody can contribute to. We also invite other people who have questions or concerns to attend committee meetings and participate in problem-solving.
Potential pitfalls:
- Documentation once created is not maintained. This is a known issue for any type of documentation, especially when it’s not in constant use. Our solution is to appoint maintainers for the docs and to be sure the notifications that Confluence sends to the maintainer’s inbox are not going to be ignored whenever somebody leaves a comment.
- The difficulty of organising retrospective meetings and getting people involved. Finding a time when all interested people are able to participate and they have no urgent tasks might be challenging. For this reason, such meetings need to be planned well in advance.
Benefits
- Less time and effort required for tests maintenance and bug fixes. Having references to guide you while writing the code makes it much clearer what to write and how to write.
- Faster code reviews. From one side the code is neater, and from the other side, there’s always a source of truth both the reviewer and reviewee can refer to.
- No extra effort required when working with code written by other teams. Since we have multiple teams who are working in the same repo, it can happen the code standards and shared practices may diverge. But having known frames help keep the code style inside the boundaries.
Share information
As a team, we are trying both to learn concepts and enhance expertise so we encourage everybody to share their knowledge with others. This could be a root cause analysis of the tricky bug; a case study; a sneak-peek into instrumentation internals; or anything considered to be potentially beneficial to the others.
This pattern can be described as SHARE INFORMATION pattern — Ask for and give information to managers, developers and other testers. For us, this is the way to benefit from a wider pool of knowledge that will improve Test Automation.
Thus, we’ve launched weekly QA lightning talks. Any person can suggest a topic and present it for 10–15 mins, with Q&As to follow. Because meetings are brief and time-limited, there’s enough time to learn something new without disrupting too much of the day. If anyone misses a presentation, there’s still a chance to catch up later, as all recordings are stored in the internal library. Lightning talks are not limited to Testing and Test Automation only but can be on a broader topic. For instance, once our colleague talked about their experience of being a Homebrew maintainer. One of the topics I personally have presented was about me being a volunteer ‘armchair mapper’ for Humanitarian OpenStreetMap. What I do there is create maps for the poorly mapped areas. These later get used by various organisations such as British Red Cross or Medecins Sans Frontieres in their fieldwork. A shortened version of this talk was later featured in the SoapBox session at the Eurostar 2020 conference.
Potential pitfalls:
- Not everybody will be ready to prepare and present their talks. Indeed, not only does it take time to prepare a presentation, but the presentation to others also requires extra effort and courage. We tackle this problem by restricting the length of talks and offering plenty of help and support to anybody interested in being a speaker. We’ve already got a bunch of people who have made their first steps in speaking.
- It will be hard to keep people interested and the attendance numbers up. We vary talk topics weekly and concentrate on the most current issues. For instance, when we integrated a table with Teamcity test runs into JIRA tickets, the Test Automation engineer who had co-developed the solution, delivered a talk on it.
Benefits:
- Improve work visibility among the team members. The point is that, occasionally, and this may sound familiar to the companies with many teams, we might inadvertently end up ‘working in silos’. Therefore, lightning talks can be a valuable addition to the actions you take towards developing a more collaborative way of working.
- Increase in the team’s general knowledge. According to the study “The Benefits of Knowledge Management: Some Empirical Evidence”, knowledge sharing improves intradepartmental and interdepartmental employee communication.
- Additional motivation for self-development and growth. When you see what’s being developed around you or where you could potentially make a difference — it may encourage you to enter a new learning domain or boost your existing level of knowledge.
Bonus: Pair up
One further point I’d like to mention is something that resulted from the previous steps that we’ve taken. Namely, one outcome was that QAs have become ready to take (and gradually taken) ownership of writing, maintaining and analysing the failed automation tests.
So, the intention now was to go a step further — with QAs taking over the Test Automation Framework and some tools related tasks, while the Test Automation team moves towards more Research, Innovation, and building internal testing tools. Such a shift would be impossible if QAs failed to learn the framework, test inner specifics or did in-depth work with the codebase — beyond their day-to-day tasks. Regular activities were being done to a decent standard, but it seemed that going beyond them would not be easily achievable without help.
To achieve our objective we’ve used the PAIR UP Process pattern — less experienced team members pair up with more experienced team members.
The result is that we’ve launched Internal Mentorships for QA Engineers within the Test Automation team. What exactly is it? QA engineers join the Test Automation team for one sprint, usually a 2-weeks period, and work on the tasks from the Test Automation team backlog, not on their own, but with help and mentoring from a Senior Test Automation Engineer. The main idea behind internal mentoring — is learning. To highlight this, we’ve devised the following manifesto which listed all goals and expectations:
- Focus on learning over strict time-bound delivery
- Interactive discussions over the temptation to work in silos
- Early and regular feedback over the end-of-mentorship retro
While there is value in the items on the right, we value the items on the left more.
More information on internal Mentorships, why and how exactly we do them, can be found in my recent talk here.
Summary
In conclusion, to resolve a commonly occurring problem in software development it’s usually a good idea to investigate patterns in the software design. As for us, by adhering to the Process patterns for Test Automation we were able to resolve the following challenges:
- Reluctance to seek aid from peers (solved using ‘Ask for help’ pattern). As Seretta Gamba and Dorothy Graham point out in their book “A Journey through Test Automation Patterns: One team’s adventures with the Test Automation“ — “Don’t be afraid to ask for help: most people actually enjoy helping”;
- High maintenance costs of the test codebase (solved with ‘Set standards’ pattern). If you have long-lasting automation and/or large-scale automation efforts, setting standards is essential.
- Information silo (solved with ‘Share information’ pattern). Speak to other people: explaining something often leads to new ideas, for you, or for the people you are talking to.
We believe that following these patterns is crucial in making your Test Automation process efficient.
Having listed the pros and potential pitfalls of the selected approaches, we also wanted to show how you can apply the patterns.
In this article, we have only shed light on some of the Process patterns for Test automation. But we hope it will serve as an inspiration and you’ll benefit from following our recommendations.
Please don’t hesitate to share in the comment section the issues you faced in your Test Automation journey and how you resolved them. We would love to hear from you.
