Mutation Testing: Improving Code Quality with Stryker
Most of my years as a software developer, I relied mostly on code coverage to base the quality of our tests suits (and by code coverage I mean line coverage). After a few years when delving into the microservices world I’ve also been introduced to the concepts of integration tests (mostly for infrastructure validations), mock tests (validate service dependencies, which may be other microservices, authentication, among others) and architecture tests (multiple teams working on the same codebase can be messy and I’ve recently found that using these may ensure at least a mininum of coherence in our code bases). It was only recently that a colleague of mine introduced me to this concept of Mutation Tests and the convertation went something like this:
“How do we know that our test suits are any good?”
“Well… if we ensure that all of our business logic is covered by our tests suits we can’t go wrong (he knew I meant code coverage)”
“You sure about that? 😏”
“Here we go…”
He then proceeded to show me an example similar to this:
public static class VatCalculator
{
public static double CalculateFinalPrice(double productPrice)
{
if (productPrice <= 200)
{
return productPrice + (productPrice * 0.23);
}
return productPrice;
}
}And the unit tests
[TestMethod]
public void VatCalculator_ProductMoreThan200_VatFree()
{
const double productValue = 485;
var finalPrice = VatCalculator.CalculateFinalPrice(productPrice: productValue);
productValue.Should().Be(finalPrice);
}
[TestMethod]
public void VatCalculator_ProductLessThan200_VatFree()
{
const double productValue = 8.8;
var finalPrice = VatCalculator.CalculateFinalPrice(productPrice: productValue);
finalPrice.Should().Be((productValue + (productValue * 0.23)));
}
What If anyone changes the conditional from <= 200 to < 200?
public static class VatCalculator
{
public static double CalculateFinalPrice(double productPrice)
{
if (productPrice < 200)
{
return productPrice + (productPrice * 0.23);
}
return productPrice;
}
}After the bug is introduced we run the unit tests again and..
Despite having full code coverage, a bug was introduced that went undetected by the unit tests.
Introducing Mutation Tests
Let’s move through the nitty-gritty and I will show you after how we’ve implemented these tests in our codebase.
Mutation Testing is a technique that helps evaluate the quality of your test suite by injecting small changes (mutations) into your codebase. These mutations simulate faults or defects that may occur in your code in real-world scenarios. The goal is to determine if your tests can detect these injected faults and accurately identify them as failures.
Each mutation is categorized as either “killed” or “survived.” A mutation is considered killed if the corresponding test(s) fail, indicating that the defect has been identified. Conversely, if the tests pass, the mutation survives, indicating a potential gap in your test coverage.
The output of the test run is a comprehensive report showing the percentage of killed and surviving mutations. This report provides insights into the effectiveness and adequacy of your test suite.
The simpler, funniest and quite accurate descriptions for mutation tests I’ve found are as follows:
“mutation tests are a way of testing our tests”
“Who will guard the guards?” (sourced from Wikipedia)
Getting our hands dirty
Now, this is all quite interesting, but implementing a test suite consisting of mutation seemed like an impossible daunting task. Mutating our codebase and then running unit tests targeting the mutated code could be a significant effort that might deter us from using it effectively. Therefore, we started looking for a library that could assist us, specifically one that supports .NET Core. We’ve found Stryker.
Hands on
Install dotnet stryker
dotnet tool install dotnet-stryker --global --ignore-failed-sourcesAdd a stryker-config.json file to the root of your test project
with the following content:
{
"stryker-config": {
"reporters": [
"progress",
"html",
"cleartext"
],
"additional-timeout": 1000,
"project": "Tests.Calculator.csproj",
"test-projects": [ "./Tests.Calculator.csproj" ],
"concurrency": 4,
"thresholds": {
"high": 80,
"low": 70,
"break": 0
},
"mutation-level": "Complete",
"ignore-mutations": [ "string", "logical" ],
"ignore-methods": [
"*ConfigureAwait*",
"ToString",
"*HashCode*"
]
}
}I recommend checking the documentation before using this in production pipelines.
Finally you simply need to run the command
dotnet-strykerAnd you should have an ouput simmilar to this:
Hint: by passing "--open-report or -o" the report will open automatically and update the report in realtime.
[14:08:31 INF] 2 mutants got status Ignored. Reason: Removed by block already covered filter
�������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
�100.00% � Testing mutant 5 / 5 � K 4 � S 1 � T 0 � ~0m 00s � 00:00:0
6
Killed: 4
Survived: 1
Timeout: 0
Your html report has been generated at:
C:\dev\mutation-tests\mutation-tests\StrykerOutput\2023-07-18.14-08-09\reports\mutation-report.html
You can open it in your browser of choice.
[14:08:37 INF] Time Elapsed 00:00:27.6084072
[14:08:37 INF] The final mutation score is 80.00 %You will also get a mutation report like I mentioned earlier simmilar to this
Using the example provided earlier we can see that one mutant survided which means that our tests are not covering this specific scenario.
Challenges we’ve encountered
While Mutation Testing offers significant benefits, it also presented us some challenges:
Time and Resource Intensive: Mutation Testing can be a computationally expensive process, especially for larger codebases. Running a full mutation analysis might take a considerable amount of time and resources.
We’ve tackled this by running several tasks in parallel (Jenkins) each executing the mutation tests for no more than 2 ou 3 projects.
Complexity: Generating mutations and analyzing the results requires a deep understanding of the code under test and the test suite itself. It also requires manual intervention to go and check out the reports.
Conclusion
We’re only now starting to take this concept seriously in our daily work and the effective results are still to be measured. I still find this concept quite interesting and be sure to try and use it in your personal and professional codebases!
I have repository containing this example, go check it out.
If you have any feedback or questions feel free to reach out!
I hope this article has been helpful to you. Thank you for taking the time to read it.
If you enjoyed this article, you can help me share this knowledge with others by:👏claps, 💬comment, and be sure to 👤+ follow.
