Snapshot Testing Libraries for Android: Paparazzi vs Shot

Fajar Febriyan

·

Follow

Published in

astronauts-id

·

8 min read

·

Feb 29, 2024

--

3

Listen

Share

What is snapshot testing?

Snapshot testing, also known as screenshot testing, is an automated testing technique that captures screenshots of User Interface (UI) components and compares them against reference images. The idea is simple: we save a reference screenshot of a UI component in a known good state, and during testing, we capture a new screenshot and compare it to the reference. If these two images match, the test passes; if not, it fails.

Then why snapshot testing?

According to the Testing Pyramid, UI testing, also known as, end-to-end testing is a type of testing recommended to be carried out in application development. UI testing is basically a mechanism meant to test the UI aspect of any software that users come into contact with. One way to do this is with a simple concept not requiring painful scenarios which is the snapshot testing.

Snapshot testing is beneficial for various reasons:

1. Visual Regression Detection: It helps identifying unintended visual changes introduced during development.
2. Ease of Use: Writing snapshot tests is often simpler and more intuitive than crafting complex UI test scripts.
3. Cross-Platform Testing: Snapshot tests can be used to ensure consistency across various Android devices and configurations.

Let’s dive into the libraries

We chose to discuss and compare these 2 libraries because they have different methods and produce different results.

Shot

Shot is an open-source snapshot testing library developed by Pedro Vicente Gómez Sánchez. It integrates seamlessly with popular testing frameworks like JUnit and Espresso, making it a popular choice among Android developers.

Paparazzi

Paparazzi is another popular snapshot testing library developed by Cash App. It offers robust screenshot comparison and supports testing on a wide range of Android devices.

How it works?

Setup:

• Choose a snapshot testing library that suits your project requirements. Libraries like “Paparazzi” or “Shot” are commonly used for snapshot testing in Android.
• Integrate the chosen library into your Android project by adding the necessary dependencies in your build.gradle file.

Record:

• The initial image that we capture is termed the “Golden Image.”
• When we already have the “Golden Image”, we can proceed to the next step, which is Verify.

Verify:

• The library compares the state of an image under test with the Golden Image. If there are differences, it flags them as potential issues or inconsistencies.

Result:

• The test results usually indicate whether the snapshots match or if there are differences, helping developers to identify where adjustments are required.

To initiate the process of recording and verifying, you’ll need to execute specific commands in your Terminal. For instance, if you’re working with Paparazzi, you can start by running the following command:

./gradlew recordPaparazziDebug

Comparing Images Technique

Paparazzi

The code below shows us Paparazzi’s image comparison technique to determine how big the difference between two images. Look at this assertImageSimilar function and the details:

• goldenImage is the reference image that we already take as a screenshot when the record command is running
• image is the target image that Paparazzi takes while verify command run
• maxPercentDifferent is the tolerance value of the difference between two screenshots

fun assertImageSimilar(
    relativePath: String,
    goldenImage: BufferedImage,
    image: BufferedImage,
    maxPercentDifferent: Double
  ) {
    ...

    val imageWidth = Math.min(goldenImage.width, image.width)
    val imageHeight = Math.min(goldenImage.height, image.height)
    val width = 3 * imageWidth
    val deltaImage = BufferedImage(width, imageHeight, TYPE_INT_ARGB)
    val g = deltaImage.graphics

    // Compute delta map
    var delta: Long = 0
    for (y in 0 until imageHeight) {
      for (x in 0 until imageWidth) {
        val goldenRgb = goldenImage.getRGB(x, y)
        val rgb = image.getRGB(x, y)
        if (goldenRgb == rgb) {
          deltaImage.setRGB(imageWidth + x, y, 0x00808080)
          continue
        }

        // If the pixels have no opacity, don't delta colors at all
        if (goldenRgb and -0x1000000 == 0 && rgb and -0x1000000 == 0) {
          deltaImage.setRGB(imageWidth + x, y, 0x00808080)
          continue
        }

        val deltaR = (rgb and 0xFF0000).ushr(16) - (goldenRgb and 0xFF0000).ushr(16)
        val newR = 128 + deltaR and 0xFF
        val deltaG = (rgb and 0x00FF00).ushr(8) - (goldenRgb and 0x00FF00).ushr(8)
        val newG = 128 + deltaG and 0xFF
        val deltaB = (rgb and 0x0000FF) - (goldenRgb and 0x0000FF)
        val newB = 128 + deltaB and 0xFF

        val avgAlpha =
          ((goldenRgb and -0x1000000).ushr(24) + (rgb and -0x1000000).ushr(24)) / 2 shl 24

        val newRGB = avgAlpha or (newR shl 16) or (newG shl 8) or newB
        deltaImage.setRGB(imageWidth + x, y, newRGB)

        delta += Math.abs(deltaR).toLong()
        delta += Math.abs(deltaG).toLong()
        delta += Math.abs(deltaB).toLong()
      }
    }

    // 3 different colors, 256 color levels
    val total = imageHeight.toLong() * imageWidth.toLong() * 3L * 256L
    val percentDifference = (delta * 100 / total.toDouble()).toFloat()
    ...
}

Paparazzi compares two images in RGB format. At first, the same coordinates (identical x and y values) from two images were compared. If the RGB value is equal it is marked as no difference, otherwise it calculates the value of deltaR, deltaG, and deltaB (representing 3 values of RGB color: red, green, and blue) then sums them to variable delta. After the iteration of all coordinates is complete, the delta is compared to the total value (image height x image width x 3 x 256) and calculates it as a percent value.

Shot

The comparison technique in Shot is slightly different from the previous one from Paparazzi, it’s more straightforward. Look at the codes below.

• screenshotis a variable that carries information such as class name and test name. This will be useful to provide the output later.
• oldScreenshotis the golden image and newScreenshotis the target image.
• differentPixels is calculated by adding up the number of different pixel arrays from those screenshots
• Then we determine the percentage of difference by comparing it with the total length of the oldScreenshotPixelsand multiplying it by 100 to get the value in percentage.
• After that, we check whether the difference between these two image screenshots is within thetolerance(this is the same as maxPercentDifferent in Paparazzi).

  private def imagesAreDifferent(
      screenshot: Screenshot,
      oldScreenshot: ImmutableImage,
      newScreenshot: ImmutableImage,
      tolerance: Double
  ) = {
    if (oldScreenshot == newScreenshot) {
      false
    } else {
      val oldScreenshotPixels = oldScreenshot.pixels
      val newScreenshotPixels = newScreenshot.pixels

      val differentPixels =
        oldScreenshotPixels.zip(newScreenshotPixels).filter { case (a, b) => a != b }
      val percentageOfDifferentPixels =
        differentPixels.length.toDouble / oldScreenshotPixels.length.toDouble
      val percentageOutOf100        = percentageOfDifferentPixels * 100.0
      val imagesAreDifferent        = percentageOutOf100 > tolerance
      val imagesAreConsideredEquals = !imagesAreDifferent
      if (imagesAreConsideredEquals && tolerance != Config.defaultTolerance) {
        val screenshotName = screenshot.name
        println(
          Console.YELLOW + s"⚠️   Shot warning: There are some pixels changed in the screenshot named $screenshotName, but we consider the comparison correct because tolerance is configured to $tolerance % and the percentage of different pixels is $percentageOutOf100 %" + Console.RESET
        )
      }
      imagesAreDifferent
    }
  }

For example from the images below, imagine each little square is a pixel. We can count the difference pixel is 2 out of 16. So the difference percentage is 12.5%.

Implementation

Study Case:

In our quest for simplicity, while experimenting with Screenshot Testing Paparazzi, we stumbled upon an astonishing discovery. Despite our deliberate efforts to maintain simplicity, an intriguing anomaly surfaced during our meticulous observations. We found that altering the font size without any corresponding margin adjustments resulted in an unexpected outcome. Strikingly, our attempt to set a tolerance of 0.1% for detection failed to yield the anticipated results.

This serendipitous discovery has propelled us into uncharted territory, igniting a fervent curiosity to delve deeper into unraveling this mystery. The anomaly we encountered has sparked a keen interest in exploring the underlying elements responsible for this unforeseen outcome. This pursuit holds promise for uncovering valuable insights into the behavior of Screenshot Testing Paparazzi and offers an opportunity to enhance our comprehension of snapshot testing methodologies at a broader level.

This unforeseen twist in our experimentation journey has not only intrigued us but also served as a catalyst for further investigation and exploration. We are driven by a conviction that a deeper exploration of this anomaly will not only enrich our understanding of the tool at hand but also potentially refine the landscape of snapshot testing techniques as a whole.

Font Weight

Font Size

Font Color

Shape Image

Padding

Conclusion

After knowing the comparison technique and the test result from some variety of cases above, we can draw conclusions about which libraries are suitable for us to use in our projects, or we can combine them. Paparazzi have an easy way to generate the images. Paparazzi has the convenience of generating images because it doesn’t require a device or emulator to run.
However, because the image comparison results are not very sensitive when using Paparazzi (in this case we can say that Shot is more sensitive), it can be concluded that in the app development process, it is better to use Shot and run it on a local device or emulator, because it can recognize tiny different between image and golden image.

Whats Next?

After coming to a conclusion, it doesn’t mean we stop here. In fact, it clarifies our path. By knowing which library is suitable, we can take the next steps:

1. Explore more about running UI tests inside the CI’s pipeline

• Need to decide the tolerance value first
• Set up screenshot test at PR level
• Show the result of the test using Danger in Github

2. Standardize the UI test script for all of our projects

• UI test scenario documentation