Building a culture of unit testing
How my team internalized engineering excellence
Unit testing is often hailed in modern software engineering as a best practice. However, many engineers and engineering managers alike would know that to truly incorporate unit testing into a team’s development flow is not easy. If you are a senior engineer or an EM trying to influence your team to write unit/ automated tests like I was, this article for you!
Mandatory note: why do we write tests anyway?
I suppose different organizations might have slightly varying reasons in encouraging developers to write unit tests. For me, writing tests can easily be traced into one of my first principles of engineering — of any work, in fact: do it with excellence.
Excellence means that we do not write code, submit an MR, and pray that it works. It means that we, developers, ensure that our work is fit for consumption of others. An intuitive API design, for instance, is one way that engineering excellence is manifested. However, obviously our interfaces is not the thing that others run—our implementation is! And there is simply no way to ensure that an implementation works as intended except by testing it. Therefore, testing our code is a manifestation of our dedication to the craft.
It is noteworthy that unit testing is not the only form of testing, in fact the same is true even for automated testing. Manual testing is also a way of ensuring that an implementation works as intended — albeit only for the moment the code is developed, in the environment it was run. Therefore,if you have a reasonable testing environment that is not too far from production, manual testing could be perfectly fine too.
Thus, I believe developers should not be too rigid about automated testing. Obsessing over code coverage or the number of test cases written are counterproductive because such metrics are blind to the needs of the users we are serving. In enterprise development, for instance, code that models relationships, business logic, and validation often require far better test coverage than a network call (we may separately ensure that it is reliable). If you deal with code that are difficult to be fully tested automatically, do at least test it manually.
That said, in many cases, particularly in my work of backend development, automated testing does provide much better ROI than manual testing. I could do it as I code in my editor. I could do TDD. I could debug each test case quickly. I simultaneously invest in preventing future regressions. The list goes on. In such setting, engineering leaders should advocate for developers to write automated testing as a tried and true way of increasing deliverable quality.
Note — for the purposes of this article, I use the term “unit testing” and “automated testing” interchangeably, since “unit testing” is simply a more familiar term for many.
How my team built a culture of unit testing
Our team of back-end engineers are six people. Most of us are capable of writing good unit tests, one of us is even a TDD enthusiast, yet at the beginning of the team formation, few of us write any tests in our MRs. Unit test is an optional step in development, and with us normally rushing to deliver features at the end of the sprint, there is just isn’t enough incentive to write tests.
Upon a bit of reflection, I found that the reason engineers avoid writing automated tests stems from both a team’s internal and external factors. Externally, stakeholders (product, business) often see automated testing as an additional cost that slows down development. If you just pitch “Hey, this is best practice and we should do it but it might make our development take 50% more time”, you are sure to be rejected.
Internally, engineers might also be ill-equipped to write tests. In legacy codebases, writing tests is often difficult to do because many parts of the code are simply not testable and are in need of a refactoring. Some engineers might also struggle to write good tests, either due to incomplete understanding of what is important to be tested, or the lack of tooling and support for testing. For instance, some engineers thought that it is important to unit-test the repository layer — written with ORM — to ensure it produces the exact intended SQL statement through string equality. The result is a set of tests that are very difficult to write and very easy to break, even if the actual returned result is the same (think how arbitrary order of inner joins are semantically equivalent). On another incident, while pairing with another engineer, I found that a single unit test took nearly half a minute to compile on his work laptop — surely a dealbreaker.
All of these factors are challenges we faced when trying to build a culture of testing. Spoiler alert — my team got over them! I believe that our success could be attributed to several initiatives:
- Show, don’t tell. As an engineering leader trying to improve your team’s delivery quality, you have to lead by example and concrete actions.
- Make it easy to write tests. Writing unit tests does carry a cost, and there is a maximum effort threshold beyond which it is simply unviable to write tests.
- Work on stuff together to share knowledge within the team. Knowledge sharing are severely hampered when you work on silos.
Show, don’t tell
I found that asking people to attend seminars upon seminars of unit testing did not, in fact, influence them to write unit tests. I’ve also found that merely setting an OKR for a target coverage to be counterproductive, primarily because it pits engineering interest against the project’s interest — timeliness. Instead, what I found to be most impactful was when a couple senior engineers in the team started writing tests ourselves in every MR. We showed the team that we were willing to put extra time in our development to ensure a higher quality delivery and less issues later down the line.
The person most responsible for the delivery must show the team that they are willing to write tests and advocate that others do the same. It is the equivalent of saying “If we are too slow because of tests, it is on me”. Certainly if the tests increase delivery quality, it will quickly make the delivery go faster, but this will allow other engineers to not feel that they are at fault for harming the delivery timeline when they commit to writing unit tests.
The only way to go fast, is to go well.
Make it easy to write tests
In some circumstances, writing tests could be difficult. We worked on top of a legacy codebase where the code layout did not advocate solitary unit-tests very well. If you’d like to see engineers write more tests in challenging codebases, you need to make writing tests easier.
One challenge we faced was that our service-layer classes are poorly structured — resulting in a mumbo-jumbo of spaghetti code of 600 lines-long functions that has tens of dependencies. This makes writing the faster, isolated, solitary unit tests difficult without some major refactoring.
To assist in testing such modules, we opted to go up the pyramid, writing more sociable (read — sociable vs solitary tests) tests either on the service or directly on API level with a live DB dependency. These tests are slower to execute and won’t give full coverage of all edge cases, but they are faster to set up then mocking the tens of dependencies that might not be used in hot paths — your software’s core functionality. They also provide a layer of protection for the utmost core functionality, paving the way for future refactoring and solitary unit testing.
We also noted to ourselves that we do not have to fix and test for every single part of our monolith. In fact, in our last project, we managed to identify that the modules we are developing is a set of new aggregates that can be isolated from the rest of the code base.
We took the opportunity to experiment with a clean architecture layout for our new aggregates within the monolith. The result was massive — it is ten times easier to hold ourselves to a higher standard within our isolated submodule and write testable components. The cognitive load to write tests, then, became much lower because we are not distracted by the noise of the messy structure of the other modules.
Work on stuff together
Finally, we needed to ensure that the knowledge how to write good tests propagate within the team. Traditionally, our team is quite siloed in task execution: Each person takes some tasks in the beginning of the sprint and a large body of WIP is started in the beginning of the sprint. However, this arrangement severely hampers knowledge transfer, and while we do already collaborate regularly in technical planning, sprint planning, code reviews, and retrospectives — automated testing is most powerful when done as part of the coding flow, and benefit tremendously from hands-on transfer of knowledge during coding itself.
We started by assigning each person to a two or three-man unit working on a set of related stories (grouped under an epic, if you may) for maximum shared context. Then, within the unit, we intensified pair and mob programming sessions, as well as encouraged fast code reviews (< 30 mins). As a result, we saw those not used to writing unit tests are eventually influenced to also write unit tests. Pair programming and code reviews by more experienced team members further ensure that high-quality tests were written.
A culture of testing built — and the benefits we uncovered
Eventually, we reached a stage where every engineer in the team consistently wrote high-quality tests for each bit of code delivered. Business logic is tested, database queries is tested, API contract implementation is tested. The result were awesome:
- Tickets rarely bounced back from QA. This ultimately means faster delivery, as each ticket bounce has a communication overhead, expands WIP (engineers would’ve switched context to another task) and slows down the entire team.
- Engineers became more tuned to finding edge cases within the business process. A habit of testing also saw our engineers striving to understand better the business context, and frequently finding edge cases that might have been missed by the product or design team.
- Stakeholders increases their trust in the team. Well-paced high-quality regular delivery meant that our product team did not have to monitor individual ticket progress within the sprint. This, in turn, allow engineers more freedom to experiment and improve engineering practices, which further increases the delivery speed and quality. Trust is then further increased, a virtuous cycle!
To conclude — I am thankful to be part of a team that continuously learns to do better and to excel in our craft. Our success in incorporating unit testing as a development culture was not done overnight, but was slowly built over a quarter that spanned our project development and the team’s formation. Throughout the project. our newly-formed team kept getting better in testing and ensuring delivery quality. The project was delivered on time, and launched with a healthy lack of bugs. I hope you could achieve the same!
