TestOps: The team your engineering department likely needs but probably doesn’t know about
I know I’m not the only person in the world who doesn’t fit comfortably into the traditional engineering departartment organization chart. For the last 12 years of 22 years as a “sofware tester”, I have injected myself into organizations under the roles of SDET, Automation Engineer, Efficiency Engineer, etc. When I get into this role, I always work to establish a team which also doesn’t have an industry standard name so I’ve built it under the auspices of the engineering team, the QA team, Engineering Enablement/Efficiency, Test Infrastructure and Automation, and so on. The mission of this team is always the same: providing application and operational support to testers for their frameworks, harnesses, auxiliary tools and deployment pipelines.
Because this team has no industry-wide standard naming or definition, software testers usually lack the terminology and breadth-of-knowledge they need to communicate why they would need such a team. Vocabulary can help us frame problems so it is for this reason I propose the industry standardize on a name for this team: TestOps.
If someone in your team is developing tests outside of simple method validations (a.k.a unit-tests), your company will need accompanying infrastructure, networking, supporting tools, pipeline modifications and test frameworks that need to be designed, implemented and maintained by someone who understands what testers need and their ongoing operational requirements as they mature. You can hand these tasks to whoever it is who designs and builds infrastructure, networking, support tools, pipeline modifications and frameworks in your company, but what you will get back by default will be most likely be designed for developers, not for testers, and will eventually be the source of “dark debt”.
While testers and developers share some operational and application level requirements, there are also areas of significant difference stemming from the fact that 1.) test harnesses are not servers but scripts and 2.) tests are intended to only run in pre-production environments.
Only a few application testers are experienced in identifying those requirements. At the same time, most organizations have more developers than testers which gives those developers more representation. By necessity, development teams usually have people who can clearly communicate the team’s operational requirements and explain why some designs are problematic and others are optimal. On the other hand, testers don’t always have someone who is capable of advocating for them both technically and operationally nor may they even realize how their requirements are different from the developers or even care. Testers usually spend some portion of their day working around the fact that no one seems to design systems for them so much so that they because numb to the awareness that their working conditions are suboptimal. And any impediment to testers decreases their ability to their job which is to assess the risk of releasing a given feature.
Most organizations, and certainly those with more than 10 development teams, need an advocate for test who can 1.) design these aforementioned systems optimally, 2.) communicate clearly with application, system developers and architects about these systems, and 3.) maintain and enhance these systems.
At this point, some people might be saying, “Isn’t this just the job of the QA Architect?” But the problem with the word “Architect” is that is bounded by context. In most organizations, the majority of architects will be application architects, and indeed, QA Architects traditionally emphasize the “application” piece of their domain, that being the test scripts and their execution.
In most engineering organizations, architects with operational specializations usually have that specialization called out in their title (think CloudOps, DBA). But even though traditional QA Architects need to understand a little bit about most operational functions in order to do their job well, most rarely put much emphasis on the operational reliability of their systems because, unfortunately, the testers who are their customers usually aren’t overly concerned about reliability of their test execution. Unless a team is doing true continuous deployment (a.k.a push-button), testers only care that their tests pass when they need them to do so, which is typically for a few days per sprint.
As the need for continuous deployment increases, testers need systems that will help them keep pace the with intended rate of development. Building those systems requires people with knowledge in the traditional specializations in software architecture (Systems, Network, Cloud).
By calling this team TestOps, its purview will be almost intuitively understood by its name. TestOps manages not only the build, deployment and execution of automated tests but also the environment that those tests run on and all associated infrastructure and configuration. They manage auxiliary tools that teams use to track and measure quality, report results and in some cases maintain any necessary compliance with regulatory bodies. They are responsible for the choice, design and implementation of test-frameworks, in-house or other wise. They assure the proper funtion of framework implementations, a.k.a test harnesses, and images that execute them. They do all this not only from an application perspective but also an operational perspective and have contacts into DevOps, SecOps, SysOps or CloudOps. They may even encompass QA Practice and guide teams in making sure the test harnesses comply with good practice lest they start negatively impacting the testing ecosystem.
Depending on the size of the company, it may or may not need an entirely separate team at first. But as your company scales past 10 ten application teams, all with some modicum of test automation, it probably makes sense to give at least 1 or more people outside of the QA Architect the full-time job of maintaining and enhancing the operational aspects of test automation because this is too much work for the QA Architect to do by themselves.
That said, operations is not the only piece of TestOps, in the same way it is not the only piece in SecOps or CloudOps, etc. Its boundary may vary somewhat from organization to organization to include test data management, over-the-wire mocking mechanisms and guidance of their use, code-analysis, so on. That said, TestOps may even run as full-stack platform team.
The important thing to emphasize here if you don’t have this team and you don’t know if anyone is looking after the operational needs of testers in your organization, you should talk to your testers. Ask them to show you their most complicated and fragile tests and make them explain why they don’t work. If none of them can satisfactorily explain these failures deterministically and you don’t have a TestOps team, you need one.
