Snapshot testing is a great tool that can help enhance the stability of our software product. In this article, we go through different points which can help us improve how we use these types of tests with their pros and cons.
What is Snapshot testing?
Snapshot testing is a kind of regression testing. In these types of tests, we first capture the correct output in the system in the form of a binary file which is called a snapshot.
The output of the system can be the visual of a view, JSON representation, data, an object dump or any other format.
Then the next time the test is run it validates the output against this captured snapshot.
When does a snapshot test fail?
The test will fail if the two snapshots do not match. The reasons behind this can be one of the followings:
- The change is unexpected and there is a failure in the system.
- The reference snapshot needs to be changed to match the updated output in the system.
UI component as Snapshot
In this scenario, a snapshot test case renders a UI component and takes a snapshot. Then compares it to a reference snapshot file stored alongside the test (already captured snapshot).
In the example above, the snapshot test failed, because the new snapshot of the UI component in the system contains a green button.
Object Dump as Snapshot
In this scenario, we have a dump representation of an object in the system which initially captured. Then we compare the new dump in the system with the expected one which was captured before.
As we can see in the above example, we have a dump for the school register object. The snapshot test failed as it shows 7 for the present property value instead of 8 that we captured in the system before.
Note that there is no other logical calculation in this test. For example, if we expected to have 20 there as the captured snapshot for the present property before, now we expected to see the same number.
Snapshot: Assertion or Information?
When a snapshot test fails, do we need to become worried about the behaviour of the system likewise we treat the unit tests?
Let us look at an example. It has been reported by the QA team that a part of our app is not accessibility-compatible. We found that an Accessibility Identifier is missing in the card view state, so we added that and the issue get fixed.
However, after this change, our snapshot test for the card view state object failed:
The test failure does not state whether the previous or new view state is correct or not, it just highlights that something has changed.
In other words, a snapshot test only asserts that the current output of the system is the same as the output before, it does not assert which snapshot, either the previous or changed one, is the correct behaviour of the system.
Thus, we can say snapshot comparing is more a sort of information rather than an assertion tool for the correct behaviour of the system.
Test Code Coverage falsifying
If we consider the snapshot testing more as information than asserting tool as discussed above, it can potentially mislead our test code coverage if both unit and snapshot tests are in the same test target. Snapshot tests especially the UI component types can easily go through most of the code which in turn incorrectly assures you the most parts of the software are being tested.
Who are the other consumers of the snapshot tests ?
Apart from the developer itself, the information created by the snapshot changes in the system can be used by the 3 main groups:
- Code Reviewers
- Designers
- QA Engineers
Code Reviewers
Other developers in the team can review the changes in the snapshot. Then they can decide whether the changes are acceptable or intentional.
Designers
One of the potential consumers of the snapshot tests is designers! For UI component snapshot tests, they can review the changes and decide if these changes are visually acceptable and meets their criteria or not.
QA Engineers
Snapshot tests can be used as a regression testing tool by QA engineers in the teams. It can help them to identify the snapshot changes in the system compared to the latest approved one and use them in their reports.
Snapshot Testing VS UI Testing
UI (automated) testing is a different tool that goes through different scenes of the app and asserts some characteristics of the app.
The main difference is that UI tests are based on user interaction. For instance, The user might click on a button, goes to another scene, and swipe the screen or a button. At the end (or meanwhile the interactions) we assert some values in a UI test. However, on the snapshot tests, we assert a single component of the system based on the given conditions. Also, UI tests are a kind of BlackBox testing which makes the user completely different.
Benefits
Taking less time to write tests compared to Unit Tests
One of the advantages of having snapshot tests is that they take less time to write while preparing unit tests can sometimes be quite time-consuming.
Making a preview of UI components
Snapshots of the UI and views in the app can bring a source of preview of these visuals, especially in case the views are programmatic and hard to imagine by the code itself.
Avoid unexpected changes in the system output
Having a good coverage of snapshot tests in our software, helps us avoid unexpected changes in the system. Because a side-effect of a change can fail the relevant snapshot, it does not let unwanted changes go to production.
Drawbacks and Challenges
Test failure on intentional updates
The software undergoes a continuous development process and you change the visual and logic of your app all the time. One of the annoying parts of snapshot tests is that they will always fail on updates, even if the change is intentional.
Hard to use in components containing many sub-elements
When we make a snapshot test of a component in the system that contains many smaller sub-elements, every change in any of these sub-elements would cause a failure in the snapshot test which can be quite annoying.
Should use the same environment for the snapshots
When performing a snapshot test, a new snapshot created by the test must be compared using the same environment that originally took the reference. Other than that it would make discrepancies between snapshots.
For instance, if we have used iOS 14.1 while taking a reference snapshot for a UI component, we should use the same version of iOS when performing the tests later.
Hard to make deterministic tests with dynamic data
Likewise other types of software tests, snapshot tests should be deterministic. Running the same tests multiple times on a component that has not changed should produce the same results every time. This is not easy to achieve when some parts of the output change in every render, for example when a snapshot contains dates or dynamic contents.
Tightly coupled to the system output
Any changes, even any insignificant parts of the output, may cause snapshot tests to fail. Developers then must manually verify that everything is still working properly and update the snapshots.
Lack of meaningful assertion
As discussed earlier in the "Snapshot: Assertion or Information?" section, snapshot failures do not indicate anything about the expected output, just it says something has changed! Unlike unit tests, snapshot tests do not contain focused, meaningful assertions or expectations.
Require more storage
Snapshots must be stored in the system. Depending on the configuration, the snapshots might be saved into the project repository. Depending on the details or type of the snapshot being recorded, it can be quite large. Extensive use of UI component snapshots can lead to consuming dozens of megabytes of space.
Conclusion
Snapshot tests are a beneficial tool to observe the changes in the system, either by reviewing changes in visual components or a JSON dump. While they are easy to write and use and can be a source for regression testing for us, we should consider the challenges they might bring for us. They should not be considered as a replacement for the unit tests, as the assertion on snapshot tests does not indicate the correct behaviour in the system. In combination with other kinds of testing tools, they can help us have more stable software and apps.