Testers are needed to find bugs. The tester saves the software from plunging into the abyss of madness. Without a tester, it is impossible to build a quality product. Without a tester, the software will be buggy. One person saves the team a lot of time and nerves, and the company saves a lot of money.
Or?…
We conducted an experiment and tested the tester in practice. For more than half a year, we worked without a tester in the team, and now it’s time to reflect on our experience.
Who are we?
We are a team in a large fintech company, responsible for a small part of the services that run products with a billion turnover. From the front end side, we only have an admin panel. Other teams develop the rest of the fronts. These fronts call our services through the REST API or asynchronously. Let’s start.
How we lived, what we suffered
First, let me explain why we couldn’t sit quietly and start our experiment.
Initial release procedure
Before the start of the experiment, we released software in a way that was the mainstream in medium and large companies in the 2010s. I will describe it in a nutshell for understanding.
- We plan the release, make a list of tasks for the release, and as the tasks are completed, the tester checks them on the test bench.
- When we have done all the tasks in the release, we make the freeze code — for example, a release branch. Code for newer tasks no longer gets into it.
- Before the deployment of the release, the tester makes a regression, and the bugs found are fixed.
- After the deployment, the tester does smoke tests, and the team is ready to fix problems that arise after the release.
We had a release every 1–2 months.
What was wrong with that?
Do you recognize the problems of your past or current projects in these mini-stories consisting of a developer’s thought flow?
A couple of sketches from our then-life
___
Story 1:
- The release gets stuck in testing, so the release date is shifted by a week or two;
- After the release, you find a bug on the prod;
- Re-release;
- Again you need to carry it for testing;
- …
- Then from another service, they are urgently asked to add a feature to the API;
- Regress again.
Story 2:
- I did the task, uploaded the code, and took the following task;
- After three days, the tester comes — there is a bug in the task;
- Fixed a bug;
- A day later, a tester comes with another bug;
- I fixed it too;
- A day later tester comes again;
- Can’t understand the origins of the bug;
- It turns out that this is not a bug at all or a bug, but the wrong service;
- …
- And after two weeks: tomorrow is the release. They did a regression, and they found the same “bug” in the pre-prod.
____
After the release, fixes need to be done and… be released with another release. A release is a tambourine dance for several days in which the whole team participates. Someone hastily completes the last tasks of the release. Someone hastily fixes bugs on completed tasks, and the tester quickly checks all this and then regresses the entire system.
You don’t want to release often — a lot of overhead. Therefore, the customer of the task usually waits several weeks for the release instead of using a feature that is already ready.
The code can hang for months without reaching the sale. Both not-very-popular features and non-critical fixes hang. For example, once, I was sorting out a problem on prod and needed more information in the logs to understand what was happening. Added logs and uploaded. A month later, I’m analyzing a similar problem — but there are still no necessary logs.
Releases are often delayed. First, the release date is determined, but as it approaches, the release shifts several times — by a day, then another 2, then another, etc.
It’s just that someone needed more time to complete their task on time, got sick, or took a day off for a couple of days. Someone can’t throw their task through testing in any way because it constantly comes back with new and new bugs.
Releases affecting multiple services at once
The game is up if the release includes a task with changes for several services at once. Most often, changes to such tasks are not backward compatible — the new version of one service will not work with the old version of another service. Therefore, it is required to release several services at the same time. These services are often turned off, updated, and synchronously start on the updated version. That is, downtime appears, and this is not cool at all.
If, during the regression, a bug is found in one of the services, then the releases of all these services are blocked. If we find a bug in one of the services after the release, we can roll back all services simultaneously. Often it remains only to heroically fix the problem in the shortest possible time, staying after work. Every minute until the bug is fixed, the damage to users and the company grows, and the pressure on the team is physically felt. Understandably, the team is under stress, which negatively affects morale and productivity in the coming days and in the long term.
Let’s be like Google
They say that in Google, most teams work without testers. Automatic tests check services, and if a bug slips into production, then it is fixed, and tests are added. Releases are made for almost every task, and the deployment of a release does not require human participation. Everything is done in the pipeline.
It turns out that if you do not have a tester, then you are not necessarily a backward company with terrible quality and flawed processes. Maybe, on the contrary, you are an advanced company with the most modern approaches.
We thought: “What can Google do that we can’t?” And we decided to try that too.
We did this:
- Tester was transferred to another team;
- We took the regression plan that the tester made and implemented end-to-end autotests for most cases. That is, the regression from manual has become automatic;
- We added integration tests so that all the main functionality was reliably covered within each service. “Testcontainers” helped us with this. Plus, we agreed that now, when doing a task, the developer covers it with tests, checking all his code. After all, there will be nothing between his task and the production except for the tests he and other developers wrote.
- We decided that it is possible to release every day, each release is backward compatible with the previous one, and service releases do not depend on each other.
- Of course, our SRE engineer put together CI / CD pipelines for us, which brought all this to life. Autotests began to run after each completed task and checked that everything was working. We started to deploy releases based on the tag in the git at any time of the day or night and allowed everyone on the team to do this.
- Automatic regression is ok, but the tester does not only regress! Sometimes manual testing is the best solution:
- Disposable items;
- Tasks with substantial changes, such as new products;
- Features with U.I.;
- Features for multiple systems, including third-party ones.
In such cases, and in general, for safety reasons, we agreed that we could sometimes invite a tester to join our team in order to test something specific.
You guys are crazy
You probably immediately have objections and questions:
- Writing tests is not the job of a developer! The developer must write the code. To sit and check how it works is to discredit the honor and glory of the uniform!
- The developer will still not test his code adequately, only check the happy path, and there will still be a lot of bugs!
- Who will start the bugs? Also developers? To sit and fill out forms for a salary of 300k/nano sec?
- Autotests are long and expensive to write. You will always spend a lot of time on them.
- Yes, every week, your releases will bring down the prod! Do not rake up incidents!
- In short, the code will eventually be written even longer, and it will work poorly!
We had all these thoughts. It was interesting to check whether these fears are justified or whether these are all patterns and the usual fear of the unknown. In contrast, we also had positive hypotheses:
- Maybe the developers will pay more attention to quality, since after them — only the release;
- There will be more test coverage, and this will increase the overall resistance of the code to changes;
- Maybe without manual testing, we will at least be able to quickly deploy on prod, reduce overhead costs and increase deployment speed;
- Developers will not be pulled out of the flow by the tester, and they will not have to constantly return to tasks that they have already released a few days or weeks ago;
- Moreover, it’s just awesome!
In general, we were ready for a different effect. We agreed on the risks and tried. What did we actually get?
Team reflection
Let’s first consider how the team experienced it and what impressions the guys had.
I asked the team 4 questions:
- How do you like it? Did you like it without a tester (purely at the level of perception and emotions)?
- What changes have you noticed, and what has become better or worse in terms of work?
- Has the quality of the product changed? Has it become better/worse?
- Would you like to live like this and recommend it to others?
Here are some answers:
____
Developer:
The process felt simpler. I burn fewer resources.
It seems that the developers have become more responsible for implementing tasks. They began to think through negative scenarios — what to do if your code breaks something after the release.
The absence of a tester did not reduce the quality of the product in any way, but it increased the speed of the pipeline, which had a positive effect on development.
The practice is definitely good, but for the ideal, you also need a manual tester who will look for non-automated scenarios and bugs on an ongoing basis without interfering with the pipeline’s passage.
Another developer:
It was interesting to try. I did not think before that this could be ok.
At first, manual testing was smeared across the team. Now I see less of that testing. Sometimes you start to get paranoid that you upload dangerous functionality.
It’s hard here. Somehow strangely, fewer bugs pop up. I hope this is true.
I want a working (!) system with an outsourced tester.
Newly Hired Developer:
Unusual, requires more attention from the developer, and the tasks of the tester fall on the shoulders of the developer.
The only thing I noticed is a lot of time spent checking the written/changed code.
Increased pipeline transit time.
I still lack a manual tester in some cases.
Another newly hired developer:
There is some transparency in what should be the output, and there are no constant sinks, what and how it works, and why.
Compared with previous experience, there is a greater immersion in the requirements for the code, which should enrich the quality. Usually, there is a conditional confrontation between Q.A. and development. Here you feel more responsible for the product.
Manual testing still seems necessary, but for exceptional cases.
Analyst:
For trivial problems, regression, etc. — life has become more pleasant.
For complex tasks with many cases, I would like someone to check whether the implementation meets the requirements and how complete they are (but it was an issue even with a tester).
The delivery process has sped up.
Judging by the lack of negativity and metrics, it hasn’t gotten any worse.
I’m up for a tester with creative thinking for non-standard cases. Maybe part-time.
___
On average, it turns out:
- At least the experience for the team was cool;
- The code delivery process has become simpler and more transparent, and the code has started to get on the market faster;
- The quality has not subsided;
- There is a little stress from the lack of an anti-bug wall, and developers feel more responsible, but overall it’s rather positive;
- The need for a tester has significantly decreased but partially remained, mainly not for regression but for non-standard tests.
Lead’s opinion (mine)
We didn’t see any drop in quality. The metrics show that the number of bugs in production has remained the same, while the number of bugs found in test environments has unexpectedly decreased by two times. Most likely, there are fewer testing bugs because developers began to find most of the bugs during the development of a task, and secondly, because developers do not always report bugs when they notice a problem on their own on a test bench before release.
Nevertheless, the quality of the prod suited us. We had two blocking incidents in six months, but they would have happened even if there had been a tester. Their reason was a change in the infrastructure configuration in the cloud.
As for the speed of development, it has decreased, according to my estimates, by 20 percent. This is if we measure it by the time the ticket was in the “development” and “review” statuses. But here, we must also consider other time costs that used to happen after the end of development: the tester now does not come to the developers with questions and bugs days and weeks after the task is uploaded to the master. The feedback cycle is now shorter due to good test coverage. The developer covers their current task well with them, which immediately highlights most problems and allows them to be fixed even before the code goes further.
In general, this approach to testing seemed more technological and interesting, and it has a valuable grain.
However, not all of the effects we saw were positive. I want to adjust our work, exploit everything we like, and remove what interferes. We discussed and came up with something like this:
What we exploit:
- The delivery process for production has become more apparent. It is more objective and formalized, less dependent on the bus factor or the personality of someone from the team.
- The speed of code delivery to the prod and the frequency of releases has increased. I would like it to continue like this.
- Developers carefully prepare the code for integration with the main branch. They understand that after them, no person will find obvious errors and check the correctness of the work. Therefore developers are better immersed in the requirements, keep in mind the meaning and purpose of their changes, and cover them with tests.
- For code reviews, it becomes more important to review not only the code but also the tests: to see if there are enough of them and what scenarios they cover.
What we want to change:
- No one performs manual testing in specific cases. The team feels insecure when rolling out a feature requiring regular manual testing. Team members do not have enough qualifications in testing for this, and external testers do not have enough immersion in the requirements, the context of the product, and the team;
- I want to develop automated tests, improve culture, and improve approaches. I want someone on the team to do this and share the experience with other teams.
- Still, developers are more inclined to check the happy way and not think about how, with a small click of a finger in the right place, you can dump the whole house of cards. The developer writes tests for his code, and no one with an alternative opinion would specifically like to find problems in the implementation. Code review does this to a certain extent, but it’s far from the main focus.
So we realized that we still needed a tester on the team. He should practically engage with the solution for a few days a month. He could be really useful in research testing to help identify those problems that are not so obvious. He could really help to pump autotests so that their coverage grows, but at the same time, they remain fast and flexible.
Now, we are hiring one for the above mentioned purposes.
By the way, about hiring
At some point, I needed to strengthen the team with developers. I found myself in a situation. Candidates ask: “Tell us about the composition of the team and the processes.” I answer: “By the way, yes, we live without a tester, but this is not bad. We abandoned it, and in general, this is not a bug but a feature, and we are that level progressive.”
It seemed to me that potential developers would be scared off by the absence of a tester. The idea that you can live without a tester is more associated with backward offices, where shitty code, rush jobs, no coffee machine in the office, and everything else. I doubted whether they would take it positively.
Nevertheless, I filled all the vacancies with excellent specialists and decided to ask them about what impression our work scheme made at the interview:
- When they found out at the interview that there was no tester on the team, how did this affect the attractiveness of the position?
- How does it feel now after a month of work?
The answers were:
____
Developer:
1. It’s more likely not about a team without a tester (many of them), but teams writing stable tests. This is an additional cool experience. I would like to participate in such a project.
2. I feel calm in the team, and I gradually begin to get used to the fact that you can deploy at night without waking up to check the logs)
Another developer:
“WOW, I want to see this!!!” Attractiveness has greatly increased, and there was a desire to try it out.
I still need to get used to long pipelines.
____
In general, the attractiveness of the vacancy may even have grown. Impact on onboarding mixed. Need more data points :)
Here, of course, it is worth considering the survivorship bias — those who “did not buy” the idea that working like us is cool probably simply did not go to our positions.
Results
To sum up our experience, you should consider switching to automatic regression if:
- You have little U.I., or you are good at testing everything automatically;
- You want to have fast deployment, frequent releases (continuous deployment), and code ready for release (continuous delivery);
- Your team is pumped enough to keep the quality in such conditions and is ready to sign up for such an experiment;
- You already have good test coverage right now. If it’s not, it’s best to build it up first;
- You don’t have a lot of legacy. However, see the point above — you can cover your legacy with tests ;).
It’s not necessary to say goodbye to the tester completely. On the contrary, this is a good opportunity to get him out of the endless regression and allow him to pump skills in automation.
