Skip to main content

Non Functional Mobile App Testing


When you are testing mobile apps there are a number of non functional elements you need to consider (that do not apply to website testing) such as push notifications, device network issues, location services and app installation. In this post I'll cover these and explain how to test these areas.

Push notifications

Push notifications were pioneered by Apple in 2008 and this technology was subsequently adopted by Google for its Android OS and by Microsoft for its Windows Phone OS. Push notifications allow an app to deliver information  to a mobile device  without a specific request from the app. This means that the app does not need to be launched for the mobile device to get the push notification. Each operating system has their own Push Notification Service. On iOS it's called Apple Push Notification Service (APNs) and Android had Google Cloud Messaging (GCM) but this has been superseded by Firebase Cloud Messaging (FCM). Note that FCM can also be used to send push notifications to iOS devices. There are also a number of platforms e.g. Amazon Web Services, that allow push notifications to be sent to mulitple operating systems.

Push notifications don't work on a simulator so you will need an actual device to test them. You will also be prompted to accept push notifications when you install the app. You should check the scenario where you don't accept them i.e. no push notifcations are sent, and the case where you do accept them i.e. push notifications should be sent.

Some push notification test scenarios to consider:
  • User receives a notification when device is asleep (online) and app is in the background        
  • User receives a notification when device is asleep (online) and app is in the foreground        
  • User receives a notification when app is in the background and device is online
  • User receives a notification when app is in the foreground and device is online        
  • User receives a notification when app is in the background and another app is running in foreground        
  • When the user receives a push notification, there should be sound and vibration
  • When the user receives multiple push notifications, all the notifications should be displayed in the notification panel with the most recent one on top

 Consider what happens when the user taps on a notification:
  • When the user taps on a notification, the app should open at the expected screen
  • When the user taps on a notification, the push should be removed from the notification panel

 Some boundary scenarios to consider:
  • When the user is not logged onto the app, they should not receive any push notifications
  • Also consider what happens when the device is offline i.e. no WiFi or network coverage:
  • Does the Push Notification Service store and send the push notification once the device returns to network coverage?
  • Is there a time limit for this i.e. what happens when the device is offline for a long time or the push notification is no longer relevant?


Network Issues


I touched on network issues in the Push Notifications section and it's important to consider how the app is expected to behave when it is out of range i.e. no WiFi or mobile coverage. If you are in the middle of a transaction, is the app smart enough to store and then send the information back to server once it's online? It's best to speak to your app designers and developers about how they plan to handle offline scenarios and then tailor your testing to cater for them. For example, does the app actually tell the user that they are offline? This is also a good time to do exploratory testing to uncover scenarios that no-one has yet thought of. For example, what happens when you logout of the app when it is offline?


Location Services

A lot of apps use Location Services to determine the device's approximate location. As with push notifications these need to be accepted when you install the app. If your app usually uses a location service, you may want to test the scenario where the user turns off the location service. Does you app notifiy the user that the location service is off and is it still able to function without a location service? Does it prompt the user to turn on location settings? All these scenarios should be considered.

To turn off the location service on an iPad or iPhone:
  1. Open Settings, swipe down and tap on Privacy.
  2. At the top, tap Location Services.
  3. Tap the toggle next to Location Services.
  4. Tap Turn Off to turn off location services for all apps or select the particular app you are testing and turn off its location service.

 To disable location tracking on an Android device:
  1. Go to Settings.
  2. Tap on “Apps & notifications.” Open the Advanced heading and look for “App permissions.”
  3. Tap on Location. You’ll find here a list of all your current apps with a toggle showing whether they have location tracking activated. Choose the test app and toggle off the location tracking.



App installation

Another mobile specific feature to test, is how the app behaves when installed for the first time versus when it's updated. For example, when you do a fresh install all user information would normally be deleted and the user would need to register again. But when the user simply updates the app to a new version, you would generally want at least the username to be preserved so that the user can simply logon without having to register anew.

The simplest way to test these scenarios is to use an app that allows over-the-air installation and testing of mobile apps such as Testflight or App Center as this will allow you to test both scenarios. See my blog post How to install a test app onto your mobile device for further info on Testflight and App Center.

I hope this post has got you thinking about the mobile device specific test scenarios that you will need to consider when testing a mobile app.

Comments

  1. Nice and interesting post, I appreciate your hard work. keep it up…!!!Thanks for such useful information, It is true that now if you want to grow your business you will surely need the mobile app testing services for your business. But for that purpose everyone needs best mobile app testing companies.

    ReplyDelete

Post a Comment

Popular posts from this blog

Let’s stop writing automated end to end tests through the GUI

What’s the problem? I have not been a fan of Selenium WebDriver since I wrote a set of automated end-to-end tests for a product that had an admittedly complicated user interface. It was quite difficult to write meaningful end-to-end tests and the suite we ended up with was non-deterministic i.e. it failed randomly. Selenium Webdriver may be useful in very simple eCommerce type websites but for most real world products it’s just not up to scratch. This is because it’s prone to race conditions in which Selenium believes that the UI has updated when, in fact, it has not. If this happens, the automated check will fail randomly. While there are techniques for reducing these race conditions, in my experience it is difficult to eradicate them completely. This means that automated checks written with Selenium are inherently flaky or non-deterministic. Maintenance of these automated checks becomes a full time job as it is very time consuming to determine whether a failing check is actuall

How I got rid of step by step test cases

In my last blog post I told you what I think is wrong with step by step test cases. In this blog post I’ll tell you how I got rid of step by step test cases at the company I work for. When I joined Yambay about 18 months ago, the company was following a fairly traditional waterfall style development approach. They had an offshore test team who wrote step by step test cases in an ALM tool called Test Track. Over the past 18 months we have moved to an agile way of developing our products and have gradually got rid of step by step test cases. User Stories and how I use them to test Getting rid of step by step test cases didn’t happen overnight. Initially we replaced regression test cases and test cases for new features with user stories that have acceptance criteria. The key to using a user story to cover both requirements and testing is to make sure that the acceptance criteria cover all test scenarios. Often product owners and/or business analysts only cover typical scenarios. It