Selecting the Right Automation Tool for Mobile App Testing

Shukurillo Solijonov

·

Follow

Published in

Axel Springer Tech

·

12 min read

·

Apr 24, 2023

--

1

Listen

Share

In the fast-paced world of mobile app development, striking the perfect balance between feature development speed and maintaining quality can be a challenging endeavor. With the goal of optimizing our testing process and shifting the responsibility for black box UI tests from developers to QA engineers, our team at NMT (National Media and Technology) embarked on a quest to find the right automation tool for mobile app testing.

Status Quo

Current Setup Visualised via Flow & Test Pyramid:

Unit tests → Integration tests → UI Espresso tests → Code review → PR after CI runs all tests → Merge → Manual QA → Test Catalogue → Crowd/Beta testing group (optional)

This setup has proven to be highly effective, with a 99%+ crash-free rate on Crashlytics.

Challenges

• Balancing feature development speed while maintaining high quality
• Automating manual QA as much as possible

Target

a. Our objective is to limit the developer’s responsibility to unit and integration tests, allowing them to move faster and provide features.
b. Shift the responsibility for black box UI Tests to QA Engineers, further developing their skills, automating and speeding up the process.

In this blog post, we’ll share our evaluation process, the tools we considered, and our final selection, shedding light on the features and criteria that ultimately led to our decision. Finally, we will share our insights on the future of automation testing and invite your thoughts on this critical aspect of app development.

When selecting a new testing automation tool, it’s essential to prioritise features that streamline processes, improve accuracy, and enhance your team’s overall efficiency. Here are the must-have and good-to-have features you should consider when evaluating your options:

Must-have features

1. Real Mobile Device Access: To create test cases that accurately reflect real-world conditions, ensure the tool provides access to actual mobile devices, not just emulators or simulators. This enables full control over the device and allows for quick adjustments. Emulators and simulators cannot replace real devices due to accuracy and reliability factors.
2. CI/CD Integration: Seamless integration with your CI/CD pipeline helps identify bugs during the development cycle and enables efficient API test triggering.
3. Automated APK/IPA Deployment: The ability to automate APK/IPA file deployment on the testing platform saves time and minimizes manual effort.
4. Test Step Recording: Simplify test case creation by capturing every action on the device screen as test steps, enabling automated test cases without coding expertise.

5. Device Cloud Compatibility (3rd Party Cloud Platform Integration): Ensure the tool supports running tests on a device cloud like BrowserStack, especially if the platform doesn’t offer in-house mobile devices for scheduled test cases.

6. Element Inspector: Save app elements (e.g., buttons, logos, or texts) by xpath or element ID, enabling easy creation, editing, and modification of test cases with minimal effort.

7. Capabilities (pre-set functions on the devices): Capabilities are a series of key-value pairs that allow configuring the tests on BrowserStack. This feature sets Google Play Store/ App Store credentials on BrowserStack devices to test in-app purchase flows

8. Storage Options: Ensure the platform offers storage for apk files, screenshots, test elements, and more, even if storage limits apply.

9. Responsive Support: Access reliable support from the test platform team to address any issues or incidents on the testing platform.

Good-to-have features

10. Mobile Screen Mirroring: Mirror mobile device screens on desktops to interact with the device and record test cases with ease.

11. Parameter Management: Save keys and values on the platform for reuse across multiple test cases, eliminating repetitive data entry.

12. Slack Integration: Receive scheduled test case results with detailed information in Slack channels for quick overviews and prompt incident response.

Investigated Testing Tools

Disclaimer: This article reflects the situation at the time of writing. As the mobile app testing landscape evolves, some details may change over time. (Written on March 15, 2023)

As we embarked on the search for an alternative testing tool, our team thoroughly explored various options. For every automation tool, we installed the application, created test cases, and examined all accessible features to gain real-world experience. Furthermore, we engaged in conversations with representatives of different automation tools to ensure a comprehensive understanding of each tool’s features and capabilities. This approach helped our team to make a precise evaluation of the tools and to make rational decisions while considering our team’s long term objectives.

Previously, we utilized TestProject, successfully automating approximately 85% of test cases and were in the process of migrating to a reliable automation tool. Unfortunately, TestProject was discontinued by its parent company Tricentis, compelling us to seek an alternative capable of replacing TestProject and fulfilling our requirements.

1) Sofy.ai

Sofy.ai is a cloud-based test automation platform with many features that is very simple and easy to use. It provides a no-code approach to test automation, making it easy for non-technical team members to create and run tests. Dynamic element identification ensures that tests are robust and reliable, even when UI changes occur. The platform also provides with devices lab where a tester can access to a variety of devices.

However, it does not offer some must-have features, such as access to local devices and capabilities to pre-set play store credentials on devices which means for us not being able to test in-app purchases. Usually, testing the business performance of the app is one of the highest priorities and it contains a main part of the testing for all apps with in-app purchase functionality. In order to be able to test this functionality, the platform offered us the possibility to rent specific devices at an extra charge, as there was no available feature for integrating with BrowserStack either. But, this extra charge made this tool less suitable for our team.

Here is a list of pros and cons of using sofy.ai which were compared while considering this tool:

Pros:

1. CI/CD integration
2. Record the steps of testing
3. Elements inspector
4. Mobile screen mirroring
5. The test results are shown like screenshot testing, which compares the base image with the last captured image
6. Parameters Management — (Saves keys and values on the platform for reuse across multiple test cases, eliminating repetitive data entry)
7. Slack Integration

Cons:

1. Absence of support for local devices
2. The functionality to automatise purchase testing scenarios can only be achieved by renting mobile devices from the platform’s device lab, incurring extra expenses.
3. Integration with 3rd Party Cloud Platform is not possible.

In summary, the limited compatibility and additional costs were reasons why we did not choose sofy.ai as our automation testing tool.

2) Maestro

Maestro offers a user-friendly and simple interface for testers to create and run test cases with minimal effort through terminal commands. However, the setup of this tool to your machine can be a bit tricky for the first time. Its advantages include easy writing and editing of test cases using YAML, as well as the ability to record testing steps. Additionally, Maestro is free if not used with the Maestro cloud and offers CI integration, but it is only available for running on emulators.

However, to further enhance its functionality, Maestro could include features that are missing right now such as recording or screenshotting where test failures occur, improving element locators, integrating with third-party tools like BrowserStack, supporting local iOS devices, and offering screenshot testing.

Here is a summary of pros and cons:

Pros:

1. Test cases can be written and edited very easily by YAML
2. Record the steps of testing
3. Free if not using maestro cloud
4. CI integration (available only to run on emulators)

Cons:

1. Record (/screenshot) where test failed is not possible*
2. Improved locators of elements are needed
3. Integration with 3rd party cloud platform (e. g BrowserStack) is missing
4. Supporting also iOS local devices is not possible — simulators cannot replace real devices.
5. The absence of screenshot testing

* Maestro cloud offers record of test results, but the price: Monthly Cost = (# of Uploads per Month) x (# of Flows) x ($0.10 USD)

3) Eggplant

Eggplant is a software testing tool that uses image recognition technology to test the functionality and performance of software applications across multiple platforms, devices, and operating systems.

Here are some of the pros and cons, as well as features of Eggplant testing tool:

Pros:

1. Supports local devices
2. Record steps of testing. The tool has an easy-to-use interface that allows users to create and execute tests quickly and efficiently.
3. Automated APK/IPA deployment
4. CI/CD integration
5. Image recognition feature (can replace elements inspector)
6. Parameters Management — (Saves keys and values on the platform for reuse across multiple test cases, eliminating repetitive data entry)
7. Slack integration
8. The tool supports both manual and automated testing.
9. Detailed reporting and analysis of test results to help identify bugs.

When it comes to pre-set capabilities feature, it requires to have a certificate for a cloud platform of an alternative provider, called Sauce Labs and there is no possibility to integrate with BrowserStack.

Cons:

1. Missing BrowserStack integration feature : as we already have BrowserStack licences, purchasing licences for another device lab would make additional costs.
2. Without BrowserStack integration it is impossible to test in-app purchase cases which is, as aforementioned, one the most important test cases for the app.

In summary, the absence of BrowserStack integration and not being able to test in-app purchases made us not consider this tool for our testing needs.

4) Testim

Testim is an automated testing tool that enables QA teams to create, run and manage automated tests for applications. Testim is also one of the products offered by Tricentis company, originally supporting only web applications and nowadays developing a new product to support mobile applications to replace its predecessor TestProject.

This automation tool is expected to offer all of the features that TestProject had, but some of the important features are still under development and have not been released yet, one of those features is integration with 3rd party cloud platform.

As we aim to employ a testing tool that satisfies all of our demands, we could not select Testim, despite its resemblance to TestProject. Most probably, it will be a good solution in the future when all of its features have been fully developed.

5) AccelQ

AccelQ is also an AI-powered continuous testing platform that enables teams to automate testing for web, mobile, and API applications. Here are some pros and cons of using AccelQ:

Pros:

1. Access to real mobile devices
2. Integration with BrowserStack
3. Elements inspector
4. Capabilities to pre-set play store credentials for in-app purchase test scenarios
5. Parameters Management — (Saves keys and values on the platform for reuse across multiple test cases, eliminating repetitive data entry)
6. Mobile-screen mirroring

Cons:

1. Integrating into your CI&CD with auto uploading APK/API to platform and triggering UI tests is not possible
2. Running tests on local or cloud device requires to generate a special link for the apk, and use it in the platform, which creates a lot of additional steps just for running the normal test
3. There is a mandatory training at an additional cost of 7 500 Euros
4. Recording actions as test steps is not possible and every step should be written manually

Overall, AccelQ is a versatile testing tool that can help teams automate their testing processes. However, its limited integration options and additional costs made it less suitable for our team.

6) Testsigma

Testsigma is an automation testing tool that allows software testers and developers to create and run automated tests for web, mobile, and desktop applications. This tool is very user-friendly and supports cross-platform testing and test management. Its integration with CI/CD tools and also with BrowserStack makes it a good option for our testing needs.

Here are the pros of using Testsigma automation tool:

Pros:

1. Access to real mobile devices
2. CI/CD integration
3. API test trigger/APK file load
4. Record the steps of testing
5. Integration with BrowserStack
6. Elements inspector
7. Capabilities feature — pre-set play store/ app store credentials for in-app purchase test scenarios
8. Mobile screen mirroring
9. Slack Integration
10. Parameters Management — (Savess keys and values on the platform for reuse across multiple test cases, eliminating repetitive data entry)

We tested all of the available features by using 30 days of trial, and we concluded that Testsigma could be a potential substitute for Testproject in fulfilling our testing needs. Nevertheless, during the trial period, we also came across some limitations that compelled us to reassess the suitability of this tool.

Cons:

1. Running multiple tests at the same time would require obtaining an equivalent number of licenses, which can be a significant cost burden.
2. Some minor UI issues: a) every time when there is a change on the device screen, a tester needs to click on the refresh button to have the current view of the device on the desktop
3. Limited element inspector: it is not possible to insert multiple locators in each element. The possibility to insert different locators, such as xpath, ID and others, would help to differentiate and identify similar elements.

However, after conducting a thorough evaluation of all various alternative tools and analysing their available features and costs, as well as holding discussions with their representatives, we have arrived at the conclusion that a perfect solution may only be achieved with a substantial investment, which is not feasible every time for every company.

A perfect solution is the enemy of practical solution. (Jo Owen, “How To Manage”, 2015, UK)

IMG Evaluation Matrix

Summary

After careful evaluation, our QA team has chosen Testsigma to handle our automation testing processes. Despite the limitations of the tool, we believe it to be the most effective solution for our needs, as it is capable of addressing all testing scenarios and meeting our essential requirements.

Now we are all hands-on, migrating our current test cases and fully automating our test catalogue.

Stay tuned to learn about our real-life experience once we get to know TestSigma in and out — the good, the bad — and the (expected) ugly side ;)

Thanks for reading! I hope that you found useful information in this post to help you find your way through the jungle of native mobile UI testing frameworks.

Now, we are curious too:
What are your thoughts on selecting the best automation tool?

Which organization, setup and tools are you using in your native mobile development process?

How do you think automation testing will evolve or change in the future?

Drop us a comment and get in touch! We are eager to exchange our experiences with you.

A heartfelt thanks to Paul Hackenberger , Sebastian Krawczyk and Sheroziy for their invaluable input and thorough review of my article. Your contributions have truly enhanced the quality of this piece. Thank you!