Maintaining our project Detox (a library for writing end-to-end tests for mobile apps) in the last couple of years has been quite a ride. With runtime optimizations introduced for iOS through brilliant companion-projects such as DetoxSync, as well as the ever-increasing maturity of Android support — paved through the wilderness, we’ve survived quite a set of challenges.

With lot’s of room to improve — still, we are now at a point where Detox feels feature-rich, user friendly and highly stable. Yet, to this day, community developers still approach us with problems often associated not with Detox itself — but with their automated test execution environments. With that, I’m mostly referring to Android.

Admittedly, the user/project onboarding experience for Detox-Android, in its own, is still no joyride. Fortunately, we have ideas of how to improve that (i.e. by introducing presets and improved bootstrap automations). Nevertheless, the environment setup, mostly of CI machines that execute build & test automations, remains a pain-point beyond our reach.

In this guide, I will provide survival tips for setting up such CI environments — meant for regularly executing automated UI tests on Android emulators, in a stable, reliable way.

Naturally, I will be assuming Detox is the testing weapon of choice — but for the most part, the tips could suit various use cases: even for (non-React) native-app developers. BTW, if you are one, you may find Detox-Native useful.

🛠 Things you are going to need to have preinstalled on the CI machine:

• ✅ Java’s runtime
• ✅ The Android SDK (and you need to know the path where it is installed on the machine’s file-system).

If you need help with that, try looking into our environment setup guide.

Things you won’t be needing:

• ❌ Android Studio: It isn’t bad to have, but I won’t be making the assumption that it’s there; Typically, CI agents are execution environments — not development environments, hence there’s no good reason to have IDE’s on them.
• ❌ Graphical User interface: That’s right! Everything Android-testing related, A-Z, can be done on raw, GUI-less linux machines. It’s good to have a GUI, but perfectly fine if you have pure, lightweight, headless Linux boxes for the task.

Baseline

In very rough terms, CI services out there give you solutions of 2 typical flavors:

1. Flavor-1: No control over the initial environment state.
   Popular pure-SaaS solutions, such as the ones offered by CircleCI, TravisCI, Bitrise, Buildkite and the likes, are typically feature-rich, and make it easy to set up execution environments. Nevertheless, whenever a build job is run, it always starts from a state predetermined by the provider, which you have no (or very little) control over. Rather, in each job, you script your way up towards the state that you need (e.g. install missing utils, update the emulator version, etc.).
2. Flavor-2: Full control over the initial environment state.
   Alternatively, there are solutions that support local (aka on-premise) or external agent hosting through 3rd-parties, such as MacStadium. Both solutions give you full control of the machines that execute your builds (and tests), and allow yo to full set up their initial state — typically, using Virtual Machine (VM) image snapshots.

As I reckon, flavor-1 solutions are the common ones in the industry. Nevertheless, some of the things I will be covering only shine with flavor-2 solutions, where you can work your way ahead-of-time towards a highly optimized initial-state.

Note: Flavor-1 and flavor-2 are terms I’ll be mentioning often throughout the post. Please take a moment to make sure you know what they’re about 🙏🏻

Creating an Emulator

Let’s get down to business!

The cornerstone of a stable environment is a stable Android emulator 📱, and it is the most fundamental building block. I’ll be focusing on Google AVD’s (i.e. Android Virtual Devices), even though there are perfectly valid alternatives out there — such as Genymotion emulators.

In the team working on Detox, we’ve long preached that for stability, Google emulators used for automation must be based on Google-free (AKA AOSP) binary-images of Android. Largely repeating our Detox environment setup guide — I’ll be specifying how to get that done from scratch.

Survival tip #1: Use AOSP (Google-free) binary-images

Setting up an AVD

Let’s create a virtual device, with specs that would be equivalent to a Google Pixel 4 running Android 10:

Ultimately, you should now have a Pixel 1-like AVD, running a clean AOSP Android 10 (SDK 29). You know… like this:

Alas, we were aiming for an AVD with the specs of a Pixel 4, not a Pixel 1 (i.e. a larger, more pixel-dense device). For that, we will have to set the following attributes in the AVD’s config.ini file (e.g. using a simple editor such as vim):

Much better. Also, while you’re at it, it’s actually quite important to apply the following attributes as well, so as to allow for more memory (note that 576Mb is effectively the maximum heap size allowed):

vm.heapSize=576
hw.ramSize=2048

Survival tip #2: Allow for large RAM and heap sizes

🤔 …but where is the config.inifile located? On a Mac/linux, it should typically be under ~/.android/avd/Pixel_4_API_29.avd/. Nevertheless, if you are uncertain of its whereabouts, run the avdmanager list avd command again and look for the path in its output:

$ ./tools/bin/avdmanager list avdName: Pixel_4_API_29
  Device: pixel (Google)
    Path: /Users/detox/.android/avd/Pixel_4_API_29.avd
  Target: Default Android System Image
          Based on: Android API 29 Tag/ABI: default/x86_64

Voilà!

🤔 …but what about specs of other devices? If you are keen on using different specs — such as those of a Pixel 6 or even a Samsung S21, you would have to look up their screen resolution and density, specifically, and apply config.ini attributes accordingly.
One way to go about it, is to run the AVD manager on Android Studio on your local computer, and reviewing the device list on the virtual device creation screen:

Alternatively, a simple Google search may also do the trick…

Launching the Emulator

This section is mostly an optimization applicable to flavor-2 environments, where the executing machine’s state can be snapshotted. Nevertheless, I recommend reading it, even if you’re a flavor-1 person.

It’s been a bumpy ride and we haven’t even launched the emulator once. With everything laid out, it’s time to do so (carefully ⚠️), for the first time.

🤔 Why is this necessary? The main reason is that initially, emulators are launched in a cold boot way (i.e. a fully-blown Android device start-up), that takes a long time to complete. For an effective CI experience, we want to have that trimmed down as much as possible.

Fortunately, thanks to the (Google) emulators’ built-in support for snapshots — which enables quick boots rather than cold boots, that are, well… very quick (literally, take less than a second), that can be achieved using 3 simple steps:

1. Launching the emulator, then waiting for it to go online and become ~idle.
2. (Optional, but recommended) Configuring stuff in it.
3. Gracefully shutting the emulator down (⇒ having a snapshot of its state persisted in the file-system).

Consequently, future launches would quickly pop the emulator back to how it was last.

Survival tip #3 (flavor-2 environments): Prelaunch and take a snapshot of the emulator, for fast future-launches.

So, let’s go over those 3 steps, one by one.

Step 1: Launching the emulator

Step 2: Pimpin’ your emulator ride 🚗

A simple yet powerful tool is Android’s built-in ability to draw on-screen indications of UI interactions, as they take place. Here’s an illustrative example:

These indicators can be quite useful when inspecting test failures through recorded video artifacts (a core feature of Detox, BTW). As long as we’re here with the emulator powered up and all… let’s enable them!

Survival tip #4: Enable built-in on-screen indicators of user interactions (taps, swipes…)

Step 3: Shutting the emulator down (and persisting a snapshot)

An indicator of a successful snapshot update is this output in the command line:

emulator: Saving state on exit with session uptime 12345 ms

Having done all of this, it is now the right time to take a snapshot of the entire execution machine (I did say flavor-2, didn’t I?). With that, tests executed in each build job would have the necessary emulators booting up in a snap.

Suppressing ANR Dialogs with Test Butler

The achilles heel of automated Android UI testing are the system-induced ANR dialogs (Application Not Responding):

As very illustratively shown above, when ANR’s pop into existence, they appear as overlays that block the automation framework (Detox and even Espresso) from interacting with the tested app, and even from “seeing” it.
All the more so, should you decide to try to “live with them”: In the world of automation, they are not easy to detect and hard to dismiss.

🤔 But why do ANR’s appear? We can go into exact definitions, here — but it’s easier to just say that Android shows them when a certain app is… not responding! 🤷‍♀ Namely, when there have been explicit or implicit interaction attempts with it, but the app (mainly, the UI thread) is somehow blocked to the point it didn’t handle the interactions for too long. That said, in Android-P, Google have decided to have ANR’s disabled by default. Nevertheless, if you’re opting for support for older OS’s, it’s something you should take into consideration when running tests on SDK<28 emulators.

Test butler is, hands-down, an amazing library by Linkedin, which — among other things, can suppress ANR’s. It can help on two levels:

1. The launch of the emulator itself: Sometimes, if the environment is a bit sluggish, ANR’s can appear for Android’s core processes, before launching any of your apps (e.g. a dialog saying “System UI is not responding”). When this happens, your tests are doomed to fail even before you’ve even had the tested app installed on the device.
2. The launch of the app: Some apps are very busy during launch. On a sluggish environment, that can induce an ANR, even if eventually the app settle into a proper working mode.

Caveats: Test Butler only works on AOSP images. Hey, wait… lucky us!

Generally speaking, you should install and activate Test Butler in your emulator as early as possible in the test flow. However, you could only address level-1 issues (emulator launch ANR’s) in flavor-2 environments.

Survival tip #5: Set up Test Butler on your emulator, to suppress ANR dialogs

Just to make it clear — non-responsiveness is by all means a sign of architectural app bugs you generally need to address, and it should be tested (benchmarked?) and monitored regardless. Here, I’m simply making the case that suppressing ANR’s should not, nevertheless, block you from running all of your UI tests, altogether.

Time to set things up. The course of action differs a great deal whether you’re running in a flavor-1 or flavor-2 environment — so let’s separate the two.

Setting up Test Butler in flavor-2 environments

Relying on the emulator launch & snapshot-save instructions from the previous section, you should do something similar, here:

1. Launch the emulator.
2. Execute:

3. Shut the emulator down (updating the snapshot).

Setting up Test Butler in flavor-1 environments

Follow Test Butler’s instructions for setting it up in your app’s tests. You still have to download the test-butler .apk file, and install it on the emulator before installing your app. You can do that in two ways:

1. Manually — as explained above.
2. Use Detox’s out-of-the-box feature for the installation part: utilBinaryPaths. That’s the recommended way with Detox, because typically Detox takes care of launching and setting up the necessary emulators, for you.

(Optional) Get your tests to verify Test Butler is running

It may be an overkill, but since we really do not want to waste our time dealing with failures due to ANR’s, in Detox’s self-test project, we assert that the executing environment has it ready to go. This is what it typically looks like in Detox — you can do the same in your project:

Wrapping Up

As I went into great length explaining, getting an Android CI to stable-land is no easy ride — and we’ve been struggling to get there, ourselves (both with Detox’s CI, and with internal projects).

The survival tips should make a solid base for you to be able to get there too, especially if you’re using Detox. However, depending on your environment and projects’ requirements, you may have to push a bit beyond that in order to really see long-term stability.

That said, beyond all lessons and tips, for us, the main game changer was, in fact, utilizing an external emulators solution. It goes by the name of Genymotion SaaS:

Once we get beyond some ongoing refactoring changes in Detox, we will make our support for Genymotion SaaS official: documentation and everything. We’ve already rolled this integration out internally, with more than 10k minutes/month of usage, and things are very solid.

The main killer-features, for us, were:

• Preinstalled device templates, with AOSP images set out of the box.
• Emulator-state snapshotting — practically seamless, to the point you can enjoy the “premium” flavor-2 optimizations I’ve mentioned even in flavor-1 environments!
• Super easy to scale. With Detox, we are big fans of cutting down execution time using parallelization (test suites’ execution distributed across multiple, concurrently running, emulators). While CI agents were starting to choke up with 3 or 4 emulators, we found that with Genymotion we can easily spawn 10 (or more) cloud instances per build job, with no hassles.

Let’s conclude with one last tip, then:

Survival tip #6: Read my upcoming blogpost about Detox integration with Genymotion SaaS!

A special thanks to Ellinor Kwok for helping us out with Genymotion.

I hope you enjoyed this read, and you’d find all of this helpful!