Snapshot Testing. Testing the UI and Beyond (Part 1)
This posts-series aims to provide a complete overview of snapshot testing and its use cases, helping the developers understand what problems it can solve effectively and how to make it part of their software verification tools.
Part 1 is focused on mobile UI development while part 2 goes beyond UI testing and explores other rather interesting non-UI related applications. Part 3 is an iOS hands-on guide that illustrates how this can be done in practice.
What is snapshot testing?
Snapshot testing is just a form of unit testing. In unit tests, you have a function, and given some input, you explicitly assert against the expected output.
The equivalent snapshot test would look like the following and as you can see the main difference is that there is no explicit expected output.
But then, how does this work? Well, you can run a snapshot test in two modes.
- When running the test in record mode, a snapshot of the output is produced and saved as an artifact on the disk (just a simple text file).
- When running the test in verify mode a snapshot of the output is produced, it then looks on the disk for a file of the previously saved reference snapshot, it compares the two and if they differ it fails with a nice error message.
When the test fails, either the change is unexpected, or the reference snapshot needs to be updated to the new version. Finally, the snapshot artifact should be committed alongside code changes and reviewed as part of your code review process¹.
Classic assertion-based tests are perfect for testing clearly defined behavior that is expected to remain relatively stable. Snapshot tests are great for testing less clearly defined behavior that may change often. UI components often change in small and trivial ways. The copy is changed, whitespace is added, a border color is modified².
Snapshot testing is by no means a replacement of unit testing, but rather an additional test mechanism³.
Testing the UI
Ιn the mobile community the most popular form of snapshot testing is screenshot testing.
Screenshot testing
Screenshot testing is a visual regression testing method where the snapshot that you produce is an image of a part of the UI (could be a whole screen or just a single view).
Comparing screenshots
When you run the test in record mode, a reference image is saved on the disk.
Then every time you run it in verify mode, a newly generated image is produced and compared to the previously saved reference image.
If they differ, then the test fails and the failure diff would look like something like that.
It is the same concept as before when snapshotting the output of a function, but instead of text as the produced artifact you now have an image, and instead of applying text comparison you now apply pixel by pixel comparison. If you can serialize your output into a format, like text or image, that can be saved to disk and you have a way to compare and produce a diff, then you can also apply snapshot testing.
Benefits
By adding screenshot testing into our mobile project, you have already gained quite a lot.
- Trivial to write, almost effortless to update
- A fast way to verify what users see and also check how your view looks while developing (No need to fire up the simulator and navigate to a specific screen just to see how the view you are developing looks like. You just take a look at the produced image snapshot)
- They provide far more coverage than a unit test normally allows (test multiple properties at once)
- Plug them in the CI and detect UI regression errors
Device configurations
These are all nice benefits but you should not stop here. To maximize what screenshot testing has to offer, you have to make sure that you test against all possible device configurations.
Language
You can quickly test how your view looks like in other languages and produce a screenshot for each language.
Or instead of supporting multiple locales and multiple screenshots for them, you could just use one single pseudo-locale⁴ to simulate strings with increased length and see how larger texts stress your UI.
Device size
You can just as easily test how your view looks like on other devices and check, for example, if something does not fit in smaller devices.
Theme
If your app supports a dark theme, no problem... This can be tested as well.
Dynamic font size
Finally, you can also see how your view supports dynamic font size.
Device configuration matrix
If you were to test each of the above configurations in a different snapshot test, you would inevitably end up with lots of generated reference screenshots. Instead, you can group different configurations and reduce the number of configurations that need to be tested. A configuration matrix helps you define your grouping strategy.
In this example, we grouped everything in two configurations.
- The large configuration tries to be as close as possible to what the majority of the users will see (using the most common device size, theme, and font size) and possibly also matches the designs⁵.
- The small configuration tries to check if everything still looks ok in small devices, using a dark theme and an XXXLarge font size.
Exploring the domain
More importantly, screenshot testing also allows you to quickly see how the same view looks like in all different cases of your domain.
Let’s take a look at the view we have been using as an example so far. Its business purpose is to inform the user about when the market is open during the day.
Just by mocking your data, you can quickly have a look at what the view looks like when the market is closed
or when there are more than one open intervals.
With screenshot tests requiring almost no effort to set up, exploring all the different UI domain cases has never been quicker and easier.
The biggest benefits
Testing all device and domain configurations
When you combine all the above, you will end up testing multiple configurations (localisations — theme — size — dynamic font size) all at once and be done with them early in the development phase.
In that way, you would have quickly tested cases that otherwise you would have never bothered to manually recreate and test. And most importantly if the CI has your back, you also avoid UI regression errors for all cases.
The perfect pull request
As a very nice side-effect, screenshot testing will change your workflow when developing the UI and how the PR would look like in that case.
- The view can and should be developed independently. You do not need to wait for the container of the view to be implemented first.
- Thus the PR becomes quite concise, as it only contains the code for the view along with screenshots of how it will like in all possible UI cases.
- More importantly, screenshots add a missing visual dimension to the code review, thus reducing cognitive effort for the reviewer. The code reviewer no longer has to try to imagine how the UI will look like, he can actually see it.
Other interesting use cases
In the scope of the Composable Architecture, the PointFree team presented in episode 86 their solution on e2e functional tests using SwiftUI and snapshot testing. It is absolutely mind-blowing 🤯 and a must-watch.
This talk (Testing and Declarative UI’s) by Nataliya Patsovska is about combining Xcode Previews and snapshot tests. Pretty interesting as well.
People have been using screenshot testing to build and test their app’s UI design system (see Playbook for iOS and Storybook for Web)
Testing animations is also possible. We have been using it to test Lottie animations by taking snapshots of the animated view. Moreover, Stagehand, a composable type-safe animation library from Cashapp, uses snapshot testing that can also generate animated PNG files to test its functionality. Similar concepts are described in this talk.
Finally, some people use as a snapshot the difference produced between two snapshots. In this way they can isolate what changed before and after you interacted with it and test just that. Check it out here.
Tools
For iOS projects, one should check the excellent SnapshotTesting library provided by the amazing PointFree team.
Every iOS developer should also check the variety of snapshot strategies that this library supports. You will find strategies for things like CGPath, UIBezierPath, WKWebView, URLRequest and so much more. Really inspiring work by the PointFree 🥇.
For the sake of completeness, I would also like to mention the iOSSnapshotTestCase library, which is the very first library available for iOS.
For Android projects, the options are the Shot library by Karumi or the most recent Paparazzi by CashApp.
You can follow me on Twitter and LinkedIn.
References — Further reading
[2] https://benmccormick.org/2016/09/19/testing-with-jest-snapshots-first-impressions/
[3] https://www.thoughtworks.com/radar/techniques/snapshot-testing-only
[4] https://netflixtechblog.com/pseudo-localization-netflix-12fff76fbcbe
[5] As a next step, you could even produce a side-by-side comparison of the generated screenshots against the designs (exported directly from e.g. Figma). And using this to automatically augment the context of a pull requests for the code reviewer would be very nice indeed.
[6] https://www.stephencelis.com/2017/09/snapshot-testing-in-swift
[7] https://www.vadimbulavin.com/snapshot-testing-swiftui-views/
[8] https://osinski.dev/posts/snapshot-testing-self-sizing-table-view-cells/
[9] https://troz.net/post/2020/swiftui_snapshots/
Special thanks to Vagelis Koutkias for introducing screenshot testing to the team a while ago (lots of the screenshots are actually part of that very first PR), and mentioning [5] while reviewing.
Special thanks to Nikos Linakis for letting me use screenshots and terms from his initial article on Screenshot testing and always coming up with the best terminology (“the perfect pull request” is all his).
Special thanks to Natalia Chalkidou and Stelios Poulakakis for their ideas about how one device configuration should match the designs.
