Snapshot Testing. Testing the UI and Beyond (Part 2)
This is part 2 of a posts series that aims to provide a complete overview of snapshot testing and its use cases, helping the developers understand what problems it can solve effectively and how to make it part of their software verification tools.
While part 1 is focused on mobile UI development, this part goes beyond UI testing and explores other interesting non-UI related applications. For understanding what is snapshot testing and how it works, take a look at the intro of part 1. Part 3 is an iOS hands-on guide that illustrates how UI testing can be done in practice.
Beyond Testing the UI
Snapshot testing is by no means limited to testing the UI. As we will see, it has lots of other interesting use cases. By listing some examples in this part of the posts series, I will try to highlight the versatile nature of snapshot testing as a rather effortless testing tool that mainly fits well under the integration testing spectrum of the software verification methodologies. And hopefully, convince you that it can become a significant part of your testing tools.
Testing complex data structures
Testing complex data structures that have lots of properties usually requires writing lots of unit tests. This of course requires time and increases maintainability.
But if you can produce a textual representation of the complex data structure (like an HTTP request as the example shown below), you could use this as a snapshot and snapshot test it, and with a single assertion test all properties at once (the method, the URL, the headers, the body, etc).
Moreover, if the complex data structure can be easily converted to JSON (like an API response), then basically for free a JSON snapshot would also test all properties at once, avoiding even to come up with a textual representation for it.
Testing complex data formats
In a similar fashion, if you write libraries or programs that produce an output with a complex data format, snapshot testing can be proven really helpful.
Imagine that your program generates PDF files and you would like to test that. In the unit test world, you would have to write multiple assertions as the PDF is a very complex data format. Instead, you could just produce an image of the generated PDF and use that as a snapshot applying again a single assertion, while also revealing how the PDF looks like.
You can check how this Swift library for generating PDF files is actually using snapshot tests for verifying its correctness, and how people even build their own custom snapshot process for testing that their generated PDFs do not change.
The same is true for programs that produce for example XML, HTML, or even code, where in the latter case you could easily reason about the generated code by just looking at the snapshots (see here) or even use the compiler to further validate its correctness (see here). Pretty amazing stuff to be honest!
Testing log and analytics events
If you need to make sure that critical paths of the code (e.g. the login flow) are always covered by logs, you would of course need to write tests for that.
Testing with unit tests would mean asserting every log event in the flow along with its properties (valuable debug information that needs to be logged). Instead, you could avoid the hassle and just write one single snapshot test that prints all log messages, in the order they should appear, along with their respective properties.
And you also get to expose them in a very human-readable way.
Along the same lines, people are actually snapshot testing warnings and errors that their CLI tools produce (see here).
Likewise, you can snapshot test the sequence of the analytics events (along with their properties) in a business-sensitive funnel and make sure the funnel does not break⁷.
A way to explore your domain
It should be pretty evident by now that readability is an intrinsic characteristic of snapshot testing. It is also its greatest strength as its output is by far the easiest to reason about.
This is why snapshot testing is an ideal way to quickly expose and explore your domain that could otherwise remain hidden. Just take a look at how the output of a snapshot test could expose all API error logs related to a specific API endpoint in a very human-readable way.
And since it is quite often that you only discover that some extra debug information might be missing from the error logs only when already in production³, you also get to see how your logs look like before going to production and reason about them.
Also, take a look at how the output of your custom date formatter changes when the language changes. Snapshot test makes setting up something like that almost effortless.
Without the convenience of setting up a snapshot test for this, probably nobody would go the extra mile to check how dates are represented for different locales (something that users will see anyway). And that would remain hidden.
And as a nice side effect, snapshot tests’ output also acts as documentation for the developer and perhaps QA or business people.
A change detector
Snapshot tests are essentially a change detector. And as such, they are ideal for writing characterization tests⁴ (also known as Golden Master Testing) when refactoring legacy code.
When refactoring the legacy code, you want to lock the behavior to a golden master version and you are only interested in whether the output has changed and not about whether it is correct.
Thus, snapshot tests, as a low effort method that tests multiple properties at once using a single assertion, are ideal in this case. You can check here how Code Climate uses snapshot tests for writing Golden Master Testing.
They are also ideal when we want to track if something changed due to a change in an external dependency that we do not control, like an external system, or an external API or even external libraries that we would like to update from time to time.
Imagine that your program output depends on an external parser library that is currently at version 1. And we want to make sure the output remains unchanged when we update the parsing library to version 2. We can use snapshot testing to quickly capture the output when the library is at version 1, update the library and check if the output is the same and the test passes.
Testing accessibility
One of the most mind-blowing applications of snapshot testing is accessibility testing.
The AccessibilitySnapshot iOS library by CashApp produces an image snapshot of your view with all of the accessibility elements highlighted using different colors, along with an ordered list of the descriptions that VoiceOver will read for each of the elements⁶.
And if the accessibility elements of this view ever change (for example the grouping of the payment-related middle row elements breaks — see next image), the test will fail.
It is pretty clear in this case as well that snapshot tests not only protect you against regression errors but also manage to expose the domain in a very human-readable way. I cannot think of a better and faster way to reason about which elements of the screen are accessible and in what order other than a simple image. Once again 🤯 Also check this one
Potential pitfalls
Of course, there also are some pitfalls about snapshot tests that we should all be aware of.
- Effortless to update but easier to accept new snapshots with bugs. This is why snapshots need proper and thorough code review.
- Since the expected output is missing (hidden inside the snapshot artifact), the context of what you test is lost. This is why we need to write really solid test descriptions.
- They might lead to fragile tests as they test multiple properties at once and a minor change in any of them (even ones that might not be of interest) will break the test. This is why we need to keep them small.
- They increase the git repository size, especially when doing screenshot testing. Maybe git LFS could be a solution?
Tools
For iOS projects, one should check the excellent SnapshotTesting library provided by the amazing PointFree team that handles both text and iOS image snapshots.
Every iOS developer should also check the variety of snapshot strategies that this library supports. You will find strategies for things like CGPath, UIBezierPath, WKWebView, URLRequest and so much more. Really inspiring work by the PointFree 🥇.
For Kotlin projects, the options are the KotlinSnapshot by Karumi when it comes to text snapshots and the Shot library by Karumi or the most recent Paparazzi by CashApp when it comes to Android image snapshots.
For the Web, the leader and pioneer in snapshot testing is the Jest library.
You can follow me on Twitter and LinkedIn.
References — Further reading
[1] https://github.com/pointfreeco/swift-snapshot-testing
[2] https://kentcdodds.com/blog/effective-snapshot-testing
[3] Has happened to me multiple times and had to release a new version just to add some extra debug info that was missing.
[4] https://michaelfeathers.silvrback.com/characterization-testing
[5] For the sake of completeness, it is worth mentioning that some libraries also support an inline assertion mode, which avoids saving the reference snapshot on the disk and directly places it inline inside the test when the test is run in record mode. See here
[6] https://cashapp.github.io/2020-05-20/making-ios-accessibility-testing-easy
[7] https://dropbox.tech/mobile/how-we-ensure-credible-analytics-on-dropbox-mobile-apps
