Lyft currently has:

1. Lots of unit tests, something we're still trying to foster the culture around, but coverage has improved significantly over time. At this point we still run all affected tests (defined by the reverse dependency tree of the files in your change) on changes, but I'm hoping at some point we can improve PR CI time by breaking out some amount of realistic breakage potential and split out some pre-submit / post-submit tests instead.

2. Few snapshot tests, these are in a POC phase. I'm curious what folks have found to be the best workflow around these, and if they're worth the trouble. I've heard some success stories from folks who spun up a web service for diffing them and easily accepting new snapshots, but this UX seems like the most difficult productivity hurdle.

3. Few UI tests, we have a bunch of these now, mostly written by our QA teams. They aren't ever blocking on PRs but folks can run them manually if they know they could break them given their changes. I'm curious here what folks are doing to continue to scale these, and how they triage breakages, who tracks down the core issues, etc.

4. Manual tests, we block releases on manual testing at this point. We're hoping that this is reduced over time as we backfill more of our UI test cases though.

We (Spotify) have a lot of unit and integration tests (32k at the time of writing, not including the tests in our shared C++ codebase). A lot of them still run in app-hosted test bundles for legacy reasons, which we're slowly fixing. These tests are hard to cache for that reason.

We're also playing around with snapshot testing. It's been super helpful for validating views in different rendering modes (dynamic font sizing, RTL layout, etc). It's currently only in our shared UI toolkit library and we're storing them in git-lfs (via artifactory). Despite being fairly young internally, the total size of all the images is getting a little unwieldy (1600 images). It's a lot to force people to check out if they're not planning on running all of the tests. Might prefer a high-availability service that downloads/caches them on-demand instead, even if there might be the occasional network flakiness from it. We haven't been bitten yet by an Xcode upgrade that forced us to re-run all the snapshots. Bound to happen sooner or later. Will cross that bridge when we get there.

We have ~500 XCUI tests that are mostly automated replacements for what would otherwise be manual end-to-end testing. Mostly non-blocking except for a very small suite of stable smoke tests that verify that the UI testing framework is happy pre-submit.

As of Oct 2020 (when I left), Uber had:

1. Unit tests, lots of them. All business logic would be covered with these.

2. Many snapshot tests on iOS. I cannot remember the exact reason why it was iOS only, but that's where we ended up with. In general, the teams liked it, but a problem has been how the snapshots are stored in the repo, and they slow down cloning.

3. UI tests, the number fluctuating over the years. Flakyness and maintenance have been issues.

4. Sanity tests (manual tests) defined by the teams, and executed by a testing team.

5. Exploratory testing done by contractors, trying to "break" the app with unexpected ways of testing.

6. Security testing: one-offs, with specialists / vendors. I'm mentioning this as this becomes increasingly important with testing.

At Robinhood

1. Mostly unit tests

2. ~400 snapshot tests. We don't use anything special here, except a custom script that handles uploading/downloading references images from S3. We've had a few tests that randomly started failing on machines that use xcode 12.4, but we've confirmed that if they are recorded on a machine using xcode 12.4 it fixes the issue. No other issues in the last ~2 years though.

3. We have 10-15 diff/PR blocking E2E UI tests with plans to grow this to closer to 50+ over the next few months. We're investing in this heavily right now. We built a record/replay service and have a job that re-records all tests each morning, then only uses replayed traffic as diff/PR blockers. We're currently heads down on some stability work, but early results have been surprisingly positive.

4. We're experimenting with hermetic UI tests. On iOS side we've built a low level socket connection to allow the XCUITest runner and application to send messages to each other. We also launch a different UIApplication that doesn't load the full app and instead just allows us to load a single feature module to a specific screen. No networking and all tests data is mocked/preloaded/cached.

Before each release we run the full test suite + have a much larger manual QA regression test plan. We have a small time working on rethinking our testing strategy and find a more scalable way to automate more of our manual QA tests. I'll definitely have a lot more to share here over the next few months, especially around E2E and hermetic UI testing

Nordstrom:

1. ~29,800 unit tests with a 100% pass rate -- otherwise code can not be pushed to master. However, we can experience flakiness with tests that use BrightFutures -- as there's no way to fake them. There are also difficulties ensuring the test state isn't polluted by previous test runs within the same spec file. This prevents us from running tests in random order.
2. ~1,500 UI tests, using a thin wrapper around XCUITest API. These are non-blocking, but take a while to run. We have never had a 100% pass rate with UI tests due to timing issues, network stubs that aren't updated when a service changes, or the UI test steps weren't updated when a change was made to the system. We also record scenarios and store network stubs in a separate repository resulting in thousands of network stubs. It is a challenge to update test stubs when a network service changes. It can also be difficult to find the respective test that needs to be updated when a feature is changed.
3. CI which runs unit tests and UI tests at regular intervals. Build breakages or unit test run failures require that it be repaired before new code can be pushed to master.
4. We have a handful of manual tests that are ran only when necessary (and possibly a handful of others that don't fall into this category). For example, logic was changed in the module that deals with notifications.

In a perfect world, I would like to see much faster UI test runs, which is somewhat achievable in unit test land when we use KIF. It should be easy to create and maintain UI tests. Having some type of flow diagram of the entire app to understand which tests are affected by a change you make in the UI would greatly increase our productivity. For example, when you make a change in a VC it would highlight the parts of the flow that are affected. Lastly, a better testing grammar that is easy to read, write, organize, provides facilities to configure the app launch by test (feature flags, configuring signed in user, etc.), and doesn't require several seconds to initialize before a test can be ran.

Shopify Mobile (Android):

1. All tests are run on CI which is required to pass to merge a PR. CI is painfully slow (~25 mins), so most developers run relevant tests manually before opening a PR. Our builds are split up into many pipelines/modules; otherwise times would be way longer.
2. ~8400 unit tests with ~100% stability.
3. ~2300 screenshot tests with ~100% stability (except for one module with around 5 notoriously flaky orientation change tests). We use our own tool for this. Having so many screenshots was a pain point when re-theming the entire app, but hasn't been otherwise. If a developer is writing a new feature or intentionally changing the UI of a screen they will need to re-record the screenshot before merging. Screenshot diffs can be viewed in the PR with GitHub.
4. We have ~20 automation/end-to-end tests at about 98% stability. These are our most flaky tests, but also have been extremely important for catching production-only issues.

Our main pain points have been how long it takes CI to run (~11 minutes to build APK and ~15 minutes of testing) as well as failure of our BuildKite testing infrastructure in general (timeouts, network errors, etc). These infrastructure issues, the few flaky orientation tests, and legitimate failures bring our master stability down to 85%, which we'd like to improve.

At Ford for our FordPass/Lincoln Way iOS apps:

1. Heavily unit test based, somewhere around 20,000 across all our modules.

2. Lots of snapshot tests that run with our unit tests, just off of https://github.com/pointfreeco/swift-snapshot-testing.

3. Coverage is good, but there's a lot of variation we're continually adding tests for. We do TDD pretty heavily, but our apps have to deal with lots of regional and product variation based on the account so always things we need to be adding in, so our unit test numbers consistently grow.

4. Performance is decent, with a lot of optimization just within Xcode and xcodebuild it takes ~24 minutes to do a full build and run on ephemeral build nodes. No stability issues, we control for flakiness and asynchronous things pretty well.

5. No UI tests, for reasons I'll note below. Lots of manual testing. There's a certain level of mocking we can do around our connected vehicle networks, but we do have to do a fair amount of end-to-end tests with actual hardware over all sorts of protocols to communicate between app and vehicles including bluetooth and USB.

Challenges we've had and are looking to improve:

• Running tests on a per-module basis for each PR based on the dependency graph as @keith described. We feel this should be doable soon and have quite a bit of work to do on cleaning our dependency graphy so it's more horizontal to get more of these benefits.
• Running module tests one-by-one is really slow. None of our tests use a test host (we mostly avoid UIControl event loop issues by using this gist: https://gist.github.com/thedavidharris/3f3ddd5304cf23c89a959434027ee139), so in theory can run headless in parallel, and we did have this working temporarily when we were building our project with Buck, but had to abandon Buck for some internal reasons and haven't found a good way to do this with just xcodebuild.
• We mostly only test snapshot tests with English localization, and have to put in scripts to update all our snapshot tests whenever our CI pulls in new strings. We'd really like to figure out a good way to pseudolocalize this or use dev hints instead of translated values that are a bit more stable.
• UI tests ended up being so much more trouble than value, as we were having to maintain tons of server mocks, flakiness, and just general strategic questions on the right way to use UI tests. As mentioned above, they just leave too many questions: they are flaky, it's not necessarily easy to turn into a bug report to explain what went wrong and who is responsible, etc. We'll probably look to use these just to automate some of our core flows, but more as just a helper to relieve some manual end-to-end pressure.

Target

• 10,000+ unit tests. It feels extremely rare that these fail with code changes after they are originally authored, though sometimes build errors in test suites offer some signal something might have gone awry. In a recent conversation I had with a few at Target, we came to the conclusion that the vast majority of the value we get from unit tests happens when code is first written – helping a developer flush out the design and edge cases. They didn't seem to do much in terms of protecting against regressions though.

• Snapshot tests we PoC'd and put on ice a few months later. The staffing levels we had on our platform team at the time couldn't really invest what it would take to make the DevX for managing test images reasonable, similar to what others have pointed out. It's a bit of a shame because I always felt just like "just as a picture is worth 1,000 words, a screenshot comp is worth 1,000 assertions". In other words, snapshot tests account for what I feel is the biggest challenge with...

• UI Tests. We've had about 4 distinct pushes towards leveraging UI tests over the last ~7 years, each time finding it difficult to land in a sweet spot of them being both useful and maintainable. From my experience, the most value comes from just asserting that a flow can execute to completion without crashing or not finding an element, which can be great for smoke tests. Because UI tests have an exponentially slower feedback loop than unit tests, and because they are much harder to debug, ownership and maintenance is a nut we have yet to crack.

• Beta Testing through TestFlight (maxed out 10k users) paired with a crash/error monitoring tool. We release to our beta group immediately after a release branch is made and that build runs on TF for ~6-7 days before we go to the App Store. We've found the TestFlight feedback extremely useful, and have worked to "democratize" it out of App Store Connect via various ASC/Fastlane tools so not everyone needs to go fishing for an ASC account (and deal with the slow as dog ASC website). This is where we detect most of our crash and network-based issues before production.

• Manual Testing managed through a tool that stores and organizes manual test cases. Culturally, we're at an inflection point as to whether we continue this practice or not – as the time and cost for manual testing (and sometimes manual testers) is significant and can be a bottleneck.

Airbnb (iOS) as of today (March 2021) has (building on what Noah mentioned):
​

• Unit tests: ~10k unit tests that run on every PR. We bundle all of our unit tests into one big target for performance reasons. Today they run at ~15 mins in CI. We use vanilla XCTest and also support Quick and Nimble.
• Snapshot tests: ~30k screenshots testing that run on every PR. These are split into Screens (full UIViewControllers) screenshots and components (UIViews) screenshots. We run these with multiple variants (iPhone / iPad / a11y - i.e. extra large font). They take ~30 mins to run on CI. We upload these screenshots to AWS and upload them to a 3rd party service called Happo. This service compares the screenshots and gives us a nice interface for our developers to accept or reject when a screenshot is different. This check blocks PRs unless all the screenshots are the same or the developer manually accepted the changes. We have had some problems with flakiness. A common problem that we've solved at the infrastructure level was using Date() in a UI component which changes every day / month and thus the screenshot changes too. We do see some flakiness sometimes with font rendering or 1 px changes and whenever Xcode updates too but IMHO the volume of these is pretty small compared to the value we get from this. @thedrick gave a talk that goes into more detail here (starts at 16:30).
• UI Tests: We use UI Tests to refer to a test that extensively mocks its dependencies and tests the interaction of one screen with its components or the interaction between multiple screens. We have 0 tests today but we're investing in this area in 2021.
• E2E tests: We use E2E tests to refer to tests that mimic the end-user experience. That is, they start from the entry point of the app, they login and go through a complete flow. They have little to no mocking on the client side and they hit real endpoints in a close to production environment. We've attempted a couple of times to have these types of tests in the past but they've have been disabled due to either flakiness or being hard to keep up to date when the product changes. We have 0 tests today but we're investing in this area in 2021.
• Manual testing: We have a team that tests each beta version of the app before we release to the app store (usually weekly) and lets us know of any issues. They can raise release blocking issues.

Halodoc

1. ~ 4000 Unit Test cases and we measure region coverage (iOS), and branch coverage (Android). We have set up our own Jenkin pipeline which would report the region coverage (iOS) or branch coverage (Android) and upload the metrics to Kibana and break the builds if it doesn't meet the expectation of 98% of testable code. We do omit UI code. To calculate UT metrics we rely on jacoco for Android and our own custom script for iOS.

2. ~ 400 test cases in our automation pipeline consisting of UI tests. We have distributed codebase and UI tests mainly focus on testing the specific functionalities which are classified into multiple SDKs. This is mainly owned by our SDETs/QA team, updating it regularly as the new features get released and maintaining it.

3. AWS Device Farm, provides the ability to run UI tests on real devices with different configurations and OS versions.

4. Manual Testing for newer features and failed UI test cases along with reporting the NFR metrics like AppLaunch, Size, Network, and Memory usage as part of our release checklist.

5. We have automated the pen-testing framework capable of performing static, dynamic(Android only), and malware analysis for iOS & Android using MobSF.

6. We are exploring the benefits of Kotlin Multi-Platform in our UI tests on Android and iOS. Creating common SDK between the platform and providing the flexibility of extending it in platform-specific UI testing.

Hi! I wonder how are you saving snapshots? We used git-lfs with Bitbucket, but now we are moving to GitLab and looking for another solution. Any suggestions?

It seems like getting UITests to be effective at scale is quite the challenge. Can anybody speak to their physical device device testing strategy? Have you found it useful to execute UITests over a representative sample of physical devices or does the simulator suffice?

At Snap/Snapchat:

We have evolved quite a bit in our testing strategies over the years:

Around 2014-2017, we invested heavily in UI tests as a way to provide coverage that's close to the customer experience. We adopted a platform-agnostic solution with Calabash to cover all the UI testing needs with the goal of having test owners write the test once, and it'll work on both iOS and Android. This tool was mostly adopted by QA engineers to write and expand the tests, but as the feature sets on Snapchat grow with the engineering workforce, we faced scalability challenges, especially when iOS and Android feature sets diverged (some features shipped on iOS first, vice versa), maintainability/onboarding new engineers became a problem as well since calabash is written in Ruby and is not native to the development environment.

In the recent years (2018 onwards), we've shifted from Calabash to native frameworks (XCUITest for iOS and Espresso/UIAutomator for Android), as we gradually shift quality/test ownership from QA to Devs. This change really helped to exponentially scale the number of contributors and test writers. We've also beefed up our internal device lab to support all the internal testing needs on real devices. However, the investment here is still very top heavy (focusing on end-to-end UI tests), and getting coverage as early as possible (in the CI pipeline), and we've continuously battled against unreliable/flaky tests in the system as well as keeping CI pipeline fast and reliable.

To combat these challenges, in 2020, we started a program to re-shape our testing strategy to be more of like Test Pyramid shape than a Test Hourglass (yes, we're very top heavy currently), here's the rough breakdown for both iOS and Android code bases:

1. Unit Test - we have some coverage here (~20k+ tests), more on Android than iOS. Few challenges around this area are getting teams to think about writing unit testable code, along with unit tests, as part of the feature development process. Retrofitting coverage, especially around legacy code, is very challenging and risky. We treat code coverage % as a guidance rather than a hard goal, and what is NOT covered is more interesting than what is covered.

2. Snapshot/Screenshot tests (~50 tests) - we are just starting to invest in this space and don't have much coverage here at all. Most of the coverage are in the e2e tests. We're hoping to convert some of the existing e2e tests (that are testing more on the unit-level UI code) to be covered in this layer.

3. Hosted app/Feature app testing (~600 tests) - as we evolve the Snapchat app and modularize the code base to be more isolated and managed by different feature teams, we are gradually unlocking this layer of testing where we have a simple lightweight app that boots up the feature modules isolated from the other feature dependencies. We're just starting to ramp up coverage in this area as well to reduce our dependency against the e2e tests. This also allows feature teams to implement their features directly on the feature app and iterate faster.

4. E2E tests - as mentioned earlier, majority of our test coverage is in this layer (~6k tests). These tests mostly run on real devices in our in house device lab. We are currently looking at sharding some of the executions to simulators/emulators, and we're looking into ways to reduce the number of tests in this layer through educating teams and providing the framework support on alternative ways to get these coverage (items 1-3 above)

5. Manual tests - We have a team of QA engineers tackling the manual tests. There are some efforts here as well to reduce the number of manual tests through automation.

Our goal is to detect bugs as early as possible and block these bugs from merging into main branch. This puts a lot of pressure on making sure the CI pipeline has both the right amount of coverage & is able to execute fast.

Happy to discuss more and learn from you all on the best practices around testing strategies 😃

I would like to open a discussion related to e2e tests.

On which environment are you running them? Closely to the production environment would be the best, but is it an environment dedicated for testing or is it an environment also used by developers, stakeholder, etc..?

If you are running e2e tests on an existing environment (let's say preprod or dev), how can you ensure that you will not break that environment or making it unusable due to performance issues for example?

Let's say you have a test in charge of creating a new item. The assertion will be to see it in the catalog : Will you remove the item at the end of the test? Otherwise your database will just increase every time the tests are run.