How to Get Started with Mobile App Testing

Vishal Gaikar

·

Follow

Published in

Globant

·

18 min read

·

Oct 1, 2020

--

1

Listen

Share

Introduction

With 3.6 billion smartphones globally and thousands of mobile applications being pushed on app stores, engineers are finding new career opportunities in mobile app development and testing. The mobile industries are getting flooded with millions of apps. Thus, mobile app testing is becoming crucial.

What is Mobile Application Testing

Mobile app testing is the process of testing mobile applications for required quality, functionality, usability, compatibility, performance, and other characteristics.

Testing is an essential part of every software development before making it available to the customers, and with the mobile app, it has become more critical. The growing number of mobile devices is leading to massive OS fragmentations, various screen sizes, and resolutions.

Android & iOS are the most popular smartphone operating systems. There are millions of applications designed for these platforms and available for download on the Google Play and Apple App Store. The Mobile App success can be measured by the number of downloads, positive reviews, and the frequent updates and the new features with bug fixes.

Types of Mobile Apps

Mobile apps are primarily categorized into three types, based on their functionalities and the technology used for programming them:

Native Apps are explicitly coded for smartphone operating systems (OS). This includes the majority of Android and iOS devices. Being programmed for one specific platform, native apps cannot be mix-matched.

Web Apps are operated similarly to native apps but are accessed via web browsers and limited feature functionality. They are not standalone apps that require downloading and installing them onto your smartphone.

Finally, there are Hybrid Apps. They are built using multi-platform web technologies (for example HTML5, CSS, and Javascript) and then this code is embedded into a native wrapper using frameworks like PhoneGap, Xamarin, Mobile Angular UI, Ionic, etc. These are mostly web apps that operate and feel like native apps.

Difference between Mobile, Web & Desktop App

Mobile applications, by their definition, are software applications that run on portable devices operating on specific OS. These types of apps are available for download through OS-specific portals like Google Play Store and Apple app store and are installed onto users’ devices with higher-level hardware access.

Web Apps download specific parts of the program from the internet onto the devices every time they run. Google Play Instant Apps are the best examples of web applications that don’t require complete installation, manual updates, and on-device hardware horsepower. They are mostly run on servers.

Desktop Apps are standalone apps that are installed on desktop and laptop devices. These can be full-fledged like Microsoft Office Suite, Google Chrome, or can perform few functions like the calculator and calendar apps. Usually, desktop apps rely entirely on device hardware.

Mostly Used Mobile Platforms (OS)

As Windows and macOS control your desktop or laptop computer, Mobile OS controls your smartphones and tablets. When you buy any mobile device, it comes with an operating system for that specific device. This smartphone OS manages the software and hardware resources of your mobile device, similar to computer operating systems.

There are following different operating systems available for mobile phones,

• Android (Google)
• iOS (Apple)
• Bada (Samsung)
• Blackberry OS (Research in Motion)
• Windows OS (Microsoft)
• Symbian OS (Nokia)
• Tizen (Samsung)

According to the survey released by statista.com in January 2020, Android and iOS are the most popular mobile operating systems that jointly possess 99% of the global market share. Till December 2019, Android dominated the smartphone industry with 74.13% share, followed by iOS with 24.79%.

Key challenges for mobile application testing

Mobile apps should be rock-solid before being launched in the market. But there are key challenges that QA engineers might face while testing their apps.

Display Resolution & Aspect Ratio: The varieties of display resolutions — from 480p to 1600p — can be overwhelming, given there are now variable refresh rates. For testing, this means running the app on dozens of devices, starting from budget devices up to bleeding-edge flagships.

Mobile Network: There are several local standards for mobile connectivity, including cellular and WiFi. For international apps, developers must ensure their apps are compatible with every standard. Data consumption is another aspect where every region might not have the privilege of unlimited usage.

Hardware Access: iPhones are renowned for stable hardware with the latest OS updates. The same isn’t applicable for Android, where you’ll find half-a-dozen SoC vendors and manufacturers. Thus, mobile apps can’t utilize Android hardware extensively, as they must support unlimited manufacturers.

OS Versions: iOS devices are famous for getting quick and consistent OS updates for five years. Who would believe that the iPhone 7 from 2016 runs on the latest iOS14? For developers, they must support older Android versions and API throughout five years of Android developments.

Internationalization: Most of the mobile apps are designed for international markets. Testing apps for translation and regional language support is only half the story. Developers and testers should also account for regional time settings and audiences. These traits have always left developers puzzled.

Start with Mobile App Test Strategy & Plan

For winning the battle of building an overall competent app, you must be aware of the challenges and develop robust mobile app testing strategies. You’ll be benefited if you keep these essential factors in mind:

• Choosing the right mobile devices for testing is essential for ensuring your app’s performance and stability across all devices.
• Testing your app on emulators might save you from spending on hardware purchases, but you will lose on monitoring hardware stats.
• The decision of cloud testing and in-house testing depends on your requirements and what you expect from your trials.
• Manual and automated testing are two complementary methods that save you time while making the outputs more statistical.
• On-device and in-house testing offers you monitor system resource usage, like CPU temps, memory usage, and battery drainage.
• For security testing, incoming and outgoing credentials must be end-to-end encrypted, no matter what network the app is using.
• The choice of tools and methods will affect the quality and terms of mobile app deployment and updates fundamentally. You need to ensure timely and cost-efficient testing while taking into account the evolving user need.

Effective Use of Mobile Vocabulary

To use smartphones and tablets, users interact with applications with gestures. There are several different gestures (see Image 1 below).

• Tap: Briefly touch the surface with a fingertip.
• Double Tap: Rapidly touch surface twice with fingertip
• Drag: Move fingertip over the surface without losing contact
• Flick: Quickly brush the surface with a fingertip
• Pinch: Touch surface with two fingers and bring them closer together
• Spread: Touch surface with two fingers and move them apart
• Press: Touch surface for an extended period of time
• Press and Tap: Press the surface with one finger and briefly touch the surface with the second finger.
• Press and Drag: Press the surface with one finger and move the second finger over the surface without losing contact
• Rotate: Touch the surface with two fingers and move them in a clockwise or counterclockwise direction
• Open: Rapidly touch the surface twice with a fingertip.
• Select: Briefly touch the surface with a fingertip.

So, it’s always an effective idea to use the above mobile vocabulary while posting bugs on bug management tools.

Understand The Orientation — Landscape & Portrait

Smartphone devices use two types of orientation- Landscape and Portrait. The accelerometer in the smartphones is used to align the screen depending on the orientation of the device.

Mobile apps use this capability and create a better user experience by providing support to the additional layout with a simple turn of a device, and without pressing any buttons. So while testing mobile apps, one should check if the app supports both the orientations or only one, and depending on this, the number of test cases can be added to test the orientations.

For eg. The YouTube app takes advantage of the available space by adding additional functionality in landscape mode (See Image 2).

Device Fragmentation

Device fragmentation refers to the uncountable numbers of mobile devices that are active at any given time. This is quite important as app developers, and testers must ensure good user experience across all such devices. According to Statista, the number of smartphone users has crossed over somewhere around the 3.5 billion mark.

There are so many combinations of mobile devices and operating systems known as fragmentation. Mobile device fragmentation is mostly associated with Android OS as there are a large number of manufacturers who provide their own forked versions of the OS such as One UI by Samsung (for Galaxy series), OxygenOS by One Plus, MIUI by Xiaomi, Huawei EMUI.

To make mobile testing manageable, we need to consider this as a crucial point while creating a test plan (see Image 3). Due to such diversity, the tester needs to assure whether a tested application is going to work perfectly with the customer-end users’ devices. The testing can achieve this by focusing on the below points,

• Identifying the target audience and the most popular devices among the target users
• Identifying the minimum set of devices the app must support
• Making use of the device cloud services to minimize the number of physical devices needed to test in-house.

Types of Testing for Mobile Applications

There are various different effective mobile testing types including Functional testing, Security testing, Performance testing, Usability testing, Localization Testing, Installation testing, Compatibility testing, and Interruption testing. Most of them are the common types that we also follow for web testing, but the following 2 types including Compatibility testing and Interruption testing are different.

Compatibility Testing

Compatibility testing refers to checking and validating your application in order to ensure behavior remains the same across device combinations. Developers spend more and more hours into planning and developing apps, as they expect the best possible user experience across all devices.

Interruption Testing

It is a type of mobile application testing that deals with — how an application reacts to interruption and resumes to a state before the interruption. While testing different types of interruptions, we need to make sure that if the application gets interrupted it should enter into a suspended state and restart as soon as the interruption ends.

We all have interruptions in our day-to-day life. Consider a real-life example of being interrupted by a call when you are watching a video on YouTube. Some of us may notice the call, ignore it and continue watching the video, a few more might attend the call, and then resume watching the video.

However, the person’s thought while watching the video is interrupted or lost for which interruption testing helps to find out which behavior the app exhibits. The expected behavior for the above example is for the video to pause while the user attends the call and resume back when the call is over.

Below there are samples of Interruptions in Mobile Application

• Incoming and outgoing calls
• Getting the text messages or WhatsApp messages
• Push Notification
• Battery Removal
• Battery Low/Full
• Device shut off
• Putting the app into the background
• The phone screen gets lock automatically
• Alarm, Calendar events
• Low Storage, Low Memory
• Network connection loss and restore
• Sending app to background/restore it

The above list for the interrupting scenarios is not exhaustive but includes the most common scenarios.

Testing On Different OS

Now we will see how one can test the apps on the Android and iOS devices and what are the useful points to keep in mind while performing these tests.

Android OS

Introduction to Android OS

Jointly developed by Google and Open Handset Alliance, and built on the Linux kernel, Android is the mobile OS that dominates almost three-quarters of the smartphone industry. Powering smartphones, tablets and even laptops, Android is primarily designed for touchscreen mobile devices.

Installation

In order to install the Android app on your devices for testing, you need to sideload the app which is from outside of Google Play. This option is disabled by default for security reasons.

Follow the below steps to enable sideloading and install the APK file,

• Allow third-party apps. Go to Menu > Settings > Security > and check “Unknown Sources”.
• Download the application on the computer.
• Connect the device and copy the application to the device storage.
• Use the File manager to install the application.
• Android will ask you to grant permission to install the app.
• Grant the permission and it should bounce you back to the installation screen.
• The app will get installed on your smartphone.

Emulators

The Android Emulator simulates a device and displays it on your development computer. Once you’ve developed the app in Android Studio, create an Android Virtual Device (AVD) for installing and running your app.

How to create a new AVD:

1. From the ‘Your Virtual Devices’ screen of the AVD Manager, click Create Virtual Device.
2. From the ‘Select Hardware’ screen, select a category and device profile, and then click Next. You can also create a new or import a hardware profile.
3. From the ‘Select a system image’ screen, select a particular API level, and then click Next. The Recommended tab lists recommended system images.
4. In case you see the Download option next to the system image, click on it to download the system image.
5. On the Verify Configuration page check all the information, select orientation, and click Finish.

Now, from the toolbar, select your application from the debugging configs. From the target device’s drop-down menu, select the AVD that you created. Finally, click Run (see Image 4).

You can launch an app on the emulator when you run a project from the Android studio, or you can drag and drop an APK file onto the emulator to install it. Once the app is installed on the emulator, you can start testing it.

Developer Options on Android

Android has amazing hidden settings that can be accessed by enabling “Developer options’ ‘ from the device setting (see Image 5). This feature has a lot of advanced and unique features including USB debugging, Tweak Animation Settings, show CPU Usage, Background process limits, Force GPU rendering, and so on.

How to Enable Developer Options:

• Go to “Settings” of your device
• Tap on “About” or “About device” or “About Phone” option
• Scroll down and then tap “Build number” 7 times.
• Enter your PIN, Pattern, or Password to enable the Developer options menu.

That’s it. The “Developer options” menu will now appear in your Settings menu.

Capture Logs from your Android device

While testing the mobile app, there are tons of reasons to provide Logs highlighting any findings to the developers.

• Enable the USB debugging on the device from the Developer Options
• Connect your device to the computer
• Launch Android Studio.
• Click View > Tool Windows > Logcat (or click Logcat in the tool window bar).
• The Logcat window will start showing the log messages

From the top-right corner, choose ‘No Filters’, and you’ll see your Android device’s logs being gathered. Highlight the required logs and save as .log (see Image 6).

Capture Screenshots from Android devices

To capture screenshots from your Android device, first, run your app on the Android device or emulator. In the Android Studio, select View > Tool Windows > Logcat. Now, select the Android device from the drop-down on the top window. Click Screen Capture on the left side of the window.

Follow the below steps to capture the screenshots without Android studio,

• Open the screen that you want to capture.
• Depending on your phone model: Press and hold the Power + Volume down buttons at the same time for a few seconds.
• If that doesn’t work, press and hold the Power button for a few seconds. Then tap Screenshot.

That’s it. At the top of the screen, you’ll find a captured Screenshot.

Record the Video of Your Screen on Android

The main reason for you to record the screen of your app is so that you can use the video footage to attach it to the defect. With Android 10, there’s now a built-in screen recorder. You can access it using the notification bar quick settings.

Also, there are so many screen recording apps for Android available on the play store. One of the best apps is — AZ Screen Recorder. It is a stable, high-quality screen recorder for Android that helps you record smooth & clear screen videos.

You can also record videos using the Android studio. To capture the video of your screen, first, run your app on an Android device. Go to Android studio, Select View > Tool Windows > Logcat. Now, from the screen recorder settings, select Bitrate, and resolution. Click Record on the left side of the Logcat window. Once that’s done, save the final output as an MP4 file.

Monitoring CPU & RAM usage

The Android Profiler lets you understand how your app utilizes the device’s CPU, memory, network, and battery resources. To open the Android Profiler, select View > Tool Windows > Profiler. When you launch the debuggable app, the Android Profiler starts collecting profiling data and resource usage.

Enabling Touches

It is a good practice to enable touches in Android, so while recording a video for the bug a circle will appear under your finger and follows you as you move around the screen. This would be helpful for developers to see how many taps were done on the screen while reproducing the bug. You can enable this from using the steps below;

Open Settings > Developer Options > Scroll down to enable Show taps.

iOS

iOS Devices

iOS is the second most popular mobile operating system after Android. Developed by Apple Inc, iOS is only limited to Apple mobile devices (see Image 7). Over the years, Android and iOS devices are competing for higher market share. But iOS is partially focused on the premium segment with a 25% share.

iOS App Installation (iTunes & xCode)

For iOS app installation using iTunes, do the following:

• Once the iOS app build is completed, download the .ipa file.
• Open iTunes and drag & drop the .ipa file into the App Library.
• Now, connect your device to iTunes and open your device apps.
• Finally, click the install button, and then click the Sync button.

For iOS app installation using Xcode, do the following:

• Download the .ipa file once the build is completed.
• Open Xcode and select Window > Devices.
• When the device screen appears, choose the device for installing.
• Drag & Drop your .ipa file into the Installed Apps section.

iOS App Installation (OTA)

OTA (Over The Air) Deployment allows you to install your apps via HTTPS. Do the following for installing your app via OTA.

• Download the .ipa file after your app build is completed.
• Upload the .ipa file on the website you prefer.
• Upload the .plist file and make sure it’s accessible via the HTTPS.
• Create an webpage for the .plist file using <itms-services://> protocol
• After you receive the link, you’ll be prompted to install the app.

You can also install iOS apps using other tools like TestFlight or Diawi. Find below detailed steps for the same.

Install apps using TestFlight

To test beta versions of apps using TestFlight, you’ll need to accept an email or public link invitation from the developer and have a device that you can use to test.

1. Install TestFlight on your Apple iPhone.
2. Now from the invitation email, tap on the public link on your iOS device.
3. Tap on Start Testing, or tap Install or Update for the app you want to test.

Install apps using Diawi

You have to upload an IPA file to diawi and then you will get a link, open the link in safari and you will be asked to install the app. Tap on install.

Obtain iOS Logs

As mentioned earlier, it is always a good idea to provide logs in case if we find an app crash or any other critical issue. This will help developers to find the root cause of the defect.

How to take the logs and share it using your iOS Device

• Open Settings
• Tap Privacy
• Tap Analytics & Improvements
• Tap Analytics Data
• Scroll down and select the app log with the correct date
• Tap on the Share icon and share the log with the developer

How to Take iOS Logs using MAC & Windows

• Connect your iOS device and sync your device with iTunes
• On Mac, go to Finder, then choose the “Go” from the menu > “Go to Folder” option. (The keyboard shortcut is Command-Shift-G.)
• Copy-and-paste this location: ~/Library/Logs/CrashReporter/MobileDevice
• Open the folder corresponding to your device.
• On Windows 8: go to C:\Documents and Settings\\Application Data\Apple computer\Logs\CrashReporter\<your iPhone’s name>\
• On Windows Vista or 7: go to C:\Users\<USERNAME>\AppData\Roaming\Apple computer\Logs\CrashReporter/MobileDevice/<DEVICE_NAME>

iOS Simulator (How to Access & Run iOS Simulator?)

Oftentimes, you may prefer launching the Simulator without launching the app. This helps in monitoring how your app launches from the home screen of your device, or if you want to test an iOS web app in the Safari browser. To launch the iOS Simulator on xCode, follow these steps (see Image 8).

• Control-click the Xcode icon and from the following shortcut menu, select Open Developer Tool > Simulator.
• Now, select your preferred iOS device and run the Simulator. This will enable an iOS home screen, where you can simulate your app.

How To Get UDID

UDID stands for Unique Device ID. For Apple iOS devices, the UDID is the unique sequence of 40 letters and numbers. For an Apple Developer to register an iOS device for testing, the unique UDID is required first. Here’s how to get the UDID of an Apple iOS device (see Image 9).

• Launch iTunes and connect your iOS device.
• Under the Devices section on iTunes, click on your device.
• Now, click ‘Serial Number’ and this should change into the UDID.
• Finally, choose “Edit” and then “Copy” from the iTunes menu.

Other Important Points To Consider

Firebase Crashlytics

Firebase Crashlytics is Google’s real-time crash reporting program that helps you with tracking and fixing stability issues with your Android app. Firebase Crashlytics saves you from tedious troubleshooting by intelligently grouping app crashes and highlighting the reasons that might have caused them.

Selecting Right Mobile Devices for Testing

Selection of mobile devices for testing is an essential part of strategic test planning that tracks success and assures high adoption rates amongst the end-users. The smartphones and tablets landscape today is highly fragmented, spanning many devices across manufacturers and models.

While selecting the devices for testing, we need to make sure that it should support the maximum number of devices. Therefore, it is important to choose the ideal list of devices to test your mobile application.

The following are the parameters for choosing devices for mobile app testing:

• Know your mobile app users goal
• Market share and popularity
• OS & OS Version Adoption
• Manufacturers
• Screen Size and Resolution
• Hardware

Emulators

It is always an effective idea to test your mobile application on real devices, but it can be considerable investments of money. So to save on the cost, we can also test our application on Emulators and Simulators. They are easy to set up and available easily. You can make the use of the emulators to explore the behavior of your mobile application.

• Android Studio AVD
• Genymotion
• Samsung Emulators for Developers

The emulator versus real device decision also depends on where you are in the development lifecycle. Here are some pros and cons of using emulators;

• Emulator/simulator(s) are in most cases open and free software that can be very easily downloaded from the Internet and ready to be tested for.
• They are cost-effective as it is a fairly inexpensive solution.
• The emulator cannot emulate battery issues, network connectivity, and other real-time data: GPS, sensors, gestures, touch force, etc.
• It cannot emulate the performance of an app in terms of incoming calls, SMS, etc.
• Touchscreen issues cannot be emulated either
• Lastly, not all mobile apps can be tested on emulators. Apps with functionality dependent on the hardware of the devices like biometry need to be tested on real devices.

Device Cloud

Device Cloud is the mobile testing environment that allows app developers to evaluate their app’s performance remotely. This offers app developers with access to modern virtual devices, without spending on hardware.

Device Clouds with physical hardware may also provide additional insights into factors like real-time CPU temps, memory usage, and battery drainage. Find below the best 5 device farms,

• Perfecto
• SauceLabs
• AWS Device Farm
• Kobiton
• Microsoft App Center

Nowadays, most of the companies use Cloud-Based Testing platforms for testing due to the reason of the device fragmentation. With device clouds, one can get a hold of many devices with different OS versions from the different manufacturers to test their apps.

There are some challenges you might face while using device clouds including testing the apps for a performance like Battery usage, storage usage. Also, network coverage testing can’t be done easily on clouds and for that, we need a real device for testing.

With real devices, you can use the device hardware to perform many tests. For eg., if the app under test needs to use the mobile’s camera (eg. to scan barcodes or click pictures) — this cannot be done in most of the device farms.

Conclusion

In today’s world of mobile apps, mobile application testing is highly recommended and very crucial for better user experience. A bug-ridden mobile application will definitely affect your app’s success ratio as the user might start posting negative reviews on the app stores and this may also set a bad impression for your app.

To boost the in-app experience we should build a solid test strategy by keeping in mind the points mentioned in the above article including the market share of the different OS, manufacturer-owned flavors of the OS, possible test scope (performance, security, stability, etc.), and selecting right devices for the testing. This will help to achieve the maximum test coverage and hence will improve the number of downloads, customer reach, and positive reviews on the stores.

Mobile app testing is essential for enterprises to engage and retain users and make them return to the app by providing the best experience possible for every customer.