Adopting device-farm for mobile testing

Kumar Ashok

·

Follow

Published in

Tide Engineering Team

·

10 min read

·

May 28, 2020

--

Listen

Share

Introduction

The above scenarios are not alien to anyone working on mobile application development. Diversity of mobile platforms/OSes and on-demand device availability are challenging and drain the budget.

This is where leveraging a device-farm comes into play. The role of a device-farm is even more crucial for ‘mobile-first’ product-companies. Here is my team’s journey at Tide (https://www.tide.co/).

What exactly is a device-farm?

A device-farm is a cloud-hosted testing environment. This offers user-friendly access to a significant number of real devices. If needed, device-farms can have simulators & emulators. Real and virtual devices are set with pre-installed OS & specifications.

Benefits of a device-farm

Using a device-farm as part of your testing strategy brings notable benefits.

A device-farm

• Helps mobile engineers to test the suitability the app — from the perspective of an end-user.
• Boosts the confidence of releasing a product to market — due to the ability to test the app across a fleet of devices.
• Optimises test environment cost-shared access to expensive assets like real devices.
• Provides access to the testing environment from any geography in any timezone. Crucial for an organisation like us with global teams — UK, Bulgaria & India.

The decision on in-house VS third-party provider

After realising the need and benefits of a device-farm, we had two options

• Build a device-farm in-house.

or

• Avail of the services of a third-party provider of such a test environment.

We opted to go for a third-party provider for the following reasons,

1. An enterprise device-farm is a speciality — It would have taken an enormous amount of time to set up.
2. Cost-ineffective — We needed a device-farm to automate the UI tests to achieve continuous delivery. With that scope, setting up a device-farm would have been cost-prohibitive.
3. Maintenance overhead — Setting up a device-farm in-house has maintenance overhead. E.g. OS updates, update inventory per market trend, general device maintenance for hardware etc.

Tide’s journey to put in place a device-farm

Before we get too far along this journey, a quick reminder of Mike Cohn’s test automation pyramid. Sticking to the pyramid is the only way to have a scalable, fast, reliable and maintainable test suite.

The pyramid directs us to limit the amount of GUI tests. It is of crucial importance to ensure the tip of the pyramid always remains robust and fit for the purpose.

Device-farm plays a crucial role in achieving that!

Let’s roll…

We completed the trip to the device-farm in five phases.

Phase 1: Due diligence — Creating a blueprint

We didn’t want to dig for a diamond when we needed a brick.

Why

We performed due diligence to understand

• Where device-farm fits in the software development ecosystem.
• What do we expect from it e.g. parallel execution, different models etc.

How

Under the guidance of our technical coach, we invested significant time and effort in

1. Determining long term test strategy and expectation from UI test automation.
2. Understanding current and future engineering ecosystem and fitment of a device-farm in it.
3. Collaboration with the business units to define (and limit) the “tip” of the test pyramid.
4. Study the offering from different device-farm providers (remember brick vs diamond analogy?).

Outcome

• A high-level architecture of our UI test framework.
• Integration points between test framework and product development pipelines.
• Role of the device-farm in the software development ecosystem.

1. Acts as a test execution engine for automated tests.
2. Interacts with test automation development pipeline when builds are ready for testing.

• A high-level requirement for device-farm.

1. Should have real devices — Android, iOS.
2. Should have cross-browser access — Safari, Chrome, IE 11 & Edge Firefox. End to end test flows need integration between Tide mobile app and Tide web application.
3. Should have the capability to interact with Bitbucket & Bitrise pipelines.

• Baseline scope of mobile UI test automation.

1. Out of 600 regression tests, we identified 52 end to end flows for the tip of that pyramid. Collaboration with business units helped to identify those flows.
2. We consulted with the development teams and pushed remaining tests down the pyramid.

Phase 2: DIY (Do it yourself) — know limitations

It helps to reach a solution faster when you know precisely what you are missing.

Why

We tried to build a UI test automation solution ourselves to understand

1. Required components of the test automation framework.
2. Limitations of having 100% in-house solution.
3. Support needed from the development team to unblock test automation, e.g. modification in the app, custom build for testing, mocks, pipeline set up etc.

How

1. We identified a subset of end to end flows covering different aspects of the app, e.g. camera capture, selfie, QR code scanning, biometrics authentication, OTP, SMS code and so on.
2. We developed a bare bone UI automation framework with Appium, log4j, Junit, cucumber using page object design pattern.
3. We identified the blocking limitations by trying all the sample end to end flows.
4. We engaged with the development team to find a solution for as many blockers as possible.

Outcome

• We set up a test automation development pipeline. It met the architectural requirements of Phase 1 (Figure 6).
• We identified blocking limitations to evaluate capabilities of device-farm providers.

1. Communication between desktop browser and mobile device for QR code scanning should be possible.
2. Biometric authentication mocking (touch id/face id) should be possible.
3. Scanning Tide card for activation should be possible.
4. Devices should be able to receive SMS using cellular services.
5. Use of the front camera and the back camera should be possible.

Phase 3: POC (Proof of concept)

You wouldn’t want to invest in an idea until you can ‘prove’ it works.

Why

We checked solution offerings from different device-farm providers to

1. Evaluate the portability of our existing test automation framework on their platform.
2. Confirm their ability to overcome blocking imitations (identified as an outcome of DIY phase).
3. Evaluate their ability to integrate with our software development ecosystem.
4. Understand the cost of implementation.
5. Understand the ease and cost of scaling. Scope of UI test automation may increases in future — new development or execution.
6. Ensure legal and data security compliance.

How

1. Created a screening questionnaire — lab set up, test creation support, test execution engine, reporting & analytics, post-implementation support and data security.
2. Shortlisted comparable providers based on their responses to the initial questionnaire.
3. Invited shortlisted providers for an initial discussion and presentation.
4. Engaged the 2 most suitable providers to prove their solutions on our mobile apps.
5. Carried out a proof of concept with clearly defined success criteria.

• Duration: 2 weeks.
• Mode: Workshop among engineers from both sides.
• Evaluation: On each success criteria and proposed solution.
• Cost: free from one provider and paid from the second provider.

5. Presented the outcome of the proof of concept phase to the internal stakeholders.

Success criteria identified for POC

The success criteria were not defined based on the provider’s offering. We defined them to be successful in our test automation journey.

1. New member registration. Prove — capturing an image & selfie with scanning provider 1, receive SMS, read and enter OTP on the mobile app, Proceed onto the security code screen.
2. Existing membership recovery. Prove — capturing an image & selfie with scanning provider 2.
3. Activate a Tide card. Prove — Log in using biometrics, capturing and reading card image and proceed further.
4. Tide on Web login. Prove — capturing QR code from Web application in the mobile app.

Outcome

1. A report card on providers’ ability to meet the predefined success criteria.
2. The criticality of those criteria that were not met during POC.
3. The pricing model for scaling.

We selected Perfecto to provide us with device-farm. We decided on them based on the

1. The outcome of the POC.
2. Feedback of the internal stakeholders.

Perfecto is the leading testing platform for Web and Mobile apps. Perfecto provides a true test environment — thousands of device, browser and OS combinations. More about perfecto on their website — https://www.perfecto.io/

Certain evaluation points could be a differentiating factor among service providers of similar abilities

- Service provider’s drive and motivation to make POC successful.

- Transparent communication on progress and challenges faced.

- Turn around time to come up with the solution of a blocking problem.

- Technical knowledge of the workforce engaged during POC.

Phase 4: Buy & try (proof of value)

We engaged with Perfecto for a buy & try. It enabled us to extend the POC to wider objectives. We had discovered them during the proof of concept, crucial for long term success.

Why

We collaborated with Perfecto for a Proof of Value phase. We decided on a small lab configuration for this phase — 3 Android devices, 2 iPhones & 2 VMs.

The goal of this phase was to

1. Develop a better understanding of Continuous quality lab.
2. Leverage Perfecto’s platform expertise to develop a functional test automation infrastructure. This needed to be in line with our initial blueprint (Figure 6).
3. Check the platform’s ability to use test automation for continuous delivery. Objectives were — Parallel execution, speed of test execution, integration with pipelines — Bitrise & Bitbucket.
4. Check analytics capabilities for test execution monitoring & reporting.

How

• Automated smoke test pack in the POV phase.
• Resolved impediments in test automation & test execution. Perfecto engineering team helped in achieving this.
• Engaged with their product SME (Perfecto blackbelt). This blackbelt helped us in.

1. Getting platform knowledge and how to use it for optimal benefit.
2. Porting our existing automation framework to the Perfecto developed test automation framework (Quantum).
3. Building test automation development pipeline in Bitbucket.
4. Achieving parallel execution on many devices and platforms (iOS, Android, desktop web).
5. Integrating Bitrise pipeline with Perfecto continuous quality lab.

Quantum is a cross-platform test automation framework. It comes with built-in cloud integration to Perfecto for high scalability. It has advanced test reporting and root-cause analysis abilities.

We moved our test automation code to Quantum. It helped us in getting on board with the Perfecto platform faster. It only needed some refactoring instead of starting from scratch. Test automation code in a lightweight framework helped!

More about Quantum on Perfecto’s website — https://www.perfecto.io/integrations/quantum

Outcome

1. We set up a functional test automation infrastructure.
2. We automated all smoke tests and executed for iOS and Android apps.
3. We integrated Bitrise pipeline with Perfecto lab to upload app bundles.
4. We scheduled automated tests to run every night from the test automation pipeline. Automated tests execute in parallel on the devices in the device-farm.

Phase 5: Implement (steady-state model)

This is the current phase we are in after completing the proof of value phase.

Determining the right configuration for the device-farm is very important. Less number of devices may lead to resource contentions. More than the required number of devices may lead to underutilisation.

Following factors determine the right configuration

1. User base: understand the current usage and trend.
2. Daytime usage: Test authoring by engineers, test execution by development pipelines.
3. Nighttime usage: Nightly execution on development builds, nightly execution on release candidates.
4. Geography of the engineering teams and their working hours. At present, our engineering teams are in London (UK), Sofia (Bulgaria) & Hyderabad (India).
5. Cost of the configuration: Device-farm is not cheap, be wise to spend!

Challenges

• Custom reporting: We are limited with predefined reporting features which come with device-farm. Custom reporting/dashboard can be challenging if you have existing reporting requirements.
• Complexity in getting technical support: We didn’t opt for the blackbelt services. It delays getting solutions for the issues faced. E.g. code access to provider’s staff, repeating product knowledge to help support-teams, the difference in time zones etc.
• Extra security measures: Per IT security guidelines, we can’t expose our internal systems. It means more dependency on the back-end development team. We need micro-services for database operations or to access internal APIs.

Conclusion

In our journey, the recipe for success was

• Support from the management to find the most suitable solution.
• Focus on the goals we wanted to achieve using a device-farm.
• Having the right expectations from the device-farm.
• Collaboration with different teams to minimise the impact of shortcomings.
• Transparent and open communication with the service provider.

There are pros and cons of engaging with an external provider (like any other solution). Overall, we found it helpful to engage with Perfecto to speed up our test automation.