Finding Memory Leaks in Mobile Apps (Part 2: Automation)
In part one of this article, we have talked about a memory monitoring tool for mobile applications that we have created, what it looks like, and how we have visualized it. Now, in this article, we will talk about how we have automated a process for this amazing memory tool.
You can revisit the part 1 here:
Finding Memory Leaks in Mobile Apps (Part 1: Visualization)
First and foremost, we needed to build automated flows to constantly monitor and prevent new memory leaks and eliminate the existing ones as soon as possible to maintain the highest possible crash-free rate for our application.
Here are the main steps of building automation flows with the use of Memory Tool:
1. Main User Flows
As our application is fairly huge, and we have thousands of automation tests, we can’t just simply implement the memory leak tests on every test case. Therefore, we have decided to get help from our data scientists!
We have asked for our application’s main flows, the most used tools, and features that hundreds of hundreds of millions of users use whenever they open our application. In no time, one of our data scientists brought us the information that we needed, every common case in our application that hundreds of millions of users go through every day.
We have automated those common main test cases separately from our main automation test cases because those test cases are going to be used only for checking the memory behavior of the main flows of our app.
Going forward, we will be able to catch the majority of the memory issues on the development branches and fix them as early as possible to prevent OutOfMemory crashes.
2. The Execution Phases of Test Cases
We talked about the main flows above, those cases are special flows, and there will not be any functional assertion checks to check if every step is successfully completed or not. All memory test cases start from the “App Home Screen” and get the memory usage metrics before doing some actions, then execute the test case flow and will return to the “App Home Screen” and check the memory metrics again. Finally, compare the memory usage data against each other to identify any possible memory loss we had during the flow execution. The reason for starting the flows from the “App Home Screen” and returning to it to end the flow is that we wanted to have a complete cycle of actions, so views will open then close and we will be at the same point from where we have started, we chose the “App Home Screen” because it is our app’s starting point. Ideally, we should expect to have the same memory value after completing the cycle.
Before getting into Thresholds declaration for our test cases, let’s talk about how reference allocation and deallocation work both in iOS & Android!
3. Memory Allocation and Deallocation in iOS & Android
iOS:
Swift uses Automatic Reference Counting (ARC) to track and manage your app’s memory usage. ARC memory management is predictable, you know exactly when an object will be deallocated.
Every time you create a new instance of a class, ARC allocates a chunk of memory to store information about that instance. This memory holds information about the type of the instance, together with the values of any stored properties associated with that instance.
Additionally, when an instance is no longer needed, ARC frees up the memory used by that instance so that the memory can be used for other purposes instead. This ensures that class instances don’t take up space in memory when they’re no longer needed.
Android:
A managed memory environment keeps track of each memory allocation. Once it determines that a piece of memory is no longer being used by the application, it frees it back to the heap, without any intervention from the programmer. The mechanism for reclaiming unused memory within a managed memory environment is known as Garbage Collection(GC).
Garbage Collection has two goals:
- Find the data objects in a program that cannot be accessed in the future
- Reclaim the resources used by those objects
Garbage Collection’s performance is different for every device. Its functionality is based on every device’s resources. For example, a device that has 4 GB RAM, needs to call GC more often to free some space than a device that has 16 GB RAM. However, we don’t generally control when a Garbage Collection event occurs, the system handles it. It has a running set of criteria for determining when to perform garbage collection.
If you would like to read more about this topic, check Garbage collection.
Before getting into the memory calculation implementation in the test cases, let’s talk about the Thresholds!
4. Choosing Appropriate Thresholds
Now that we have our main test flows and we know how memory behaves on iOS & Android, we need to decide on a memory usage limit and pass it to our condition in test cases to terminate the testing whenever the memory value surpasses the limit.
iOS:
We have dozens of iOS devices on which our test cases will be run. The device selection for running the test cases is totally random, the test cases will be executed on different devices on our device farm. As we don’t have device-specific RAM optimization we just needed to choose a threshold for iOS devices. For example, if the application’s passive memory usage value is 100 MB, after doing some actions and returning back to the passive mode, the memory must be around 100 MB again. If the memory value does not decline to 100 MB after some actions, we can assume there may be a memory leak and start an investigation. For a starting point, we chose our threshold to be +15 MB. In our example above, the memory value starts from 100 MB. Therefore, 115 MB will be our threshold limit.
Android:
Just like iOS, we also have dozens of Android devices available for testing. The main difference with Android devices is that the RAM sizes range from 2GB to 16GB and a threshold of +15MB may be good for a device that has 2GB RAM but it may not be a good limit for a device that has 16GB RAM. We needed to test our main flows on different Android devices that have different RAM sizes to find the most optimum threshold for all devices. In addition, there is one more thing, as we discussed above the reference allocation and deallocation in Android, the Garbage Collector cannot be controlled nor decided when to be called, so, we kind of had to study and investigate how the memory behaves and when it enters a passive mode so that whenever we get the memory value we can be sure that the Garbage Collector deallocated all the unused references and we got the concrete memory usage value to do the comparison. We wanted to find an optimal way to investigate the memory’s behavior on different devices and save the memory values somewhere so that we can see the memory difference between different devices.
We have used an Android Debug Bridge(ADB) command to get the TOTAL memory value of the mobile device. This is the command:
adb shell dumpsys meminfo "appBundleID" | grep "TOTAL" | awk 'NR==1{print $2}'Then we inserted this command in a script that contains a for loop to execute the ADB command every 1 sec and prints the memory values in the file that we have created: reportValues.log
Check out the script:
echo "Started reporting memory values in reportValues.log"
> reportValues.log
i=0
for i in `seq 1 1000`;
do
adb shell dumpsys meminfo "appBundleID" | grep "TOTAL" | awk 'NR==1{print $2}' >> reportValues.log
sleep 1
doneWe run the script file using the command bash script.sh and in a new terminal we read the log by using the tail -f reportValues.log command.
Which looks like this:
It’s printing the values in KB here, we converted it to MB in the code, which I will get to it below.
Furthermore, by using this logic, we can run our flows and watch the values closely so we can understand and find the optimum threshold value for Android devices and use it for our automation tests.
After testing on a few devices, we have found that +30 MB is the best option for our threshold limit. It’s important to note that this is our starting point(+30 MB for Android and +15 MB for iOS), and we plan to gradually decrease these numbers as we gain more insights and refine our memory monitoring process.
5. Getting The Memory Value in Code
As we have decided our threshold values, let’s see how are we getting the memory values in our tests:
iOS:
We are using the memory monitoring tool’s UI to get the device’s memory(RAM) value that the application uses, let’s revisit how it looks like:
One of our developers has added an accessibility label on the tool so that we can be able to get the data in our tests and compare the memory values in each loop.
Accessibility label: A brief label in a localized string that identifies the accessibility element.
This is how we declared our accessibility label in our project:
@iOSXCUITFindBy(id = "memory_debug_tool_view")
private MobileElement memoryToolsView;And here’s the method that will get us the memory value from the tool’s UI:
public String getMemoryValue() {
return getText(memoryToolsView);
}We are getting the text that is visible on our memory tool’s UI and using it to start the memory comparison in every loop.
Android:
Unlike iOS, we tried a different approach to get the memory value in Android devices. As we have discussed above, one of the ways is with the ADB command, we could have implemented the ADB command in our project, but instead, we have used a more optimal approach which is using Appium’s native method to get the memory info of the device. As our project is using the Appium framework, we have a native method already implemented in the project, we just need to call it to receive the memory value.
What is Appium? Appium is an open source test automation framework for use with native, hybrid and mobile web apps.
It drives iOS, Android, and Windows apps using the WebDriver protocol.
As you see from the definition above, we need to get the driver to call for the method which gives us the memory value. Here’s what it looks like:
public double getMemoryValue() {
WaitUtils.threadSleep(5000); //we need ~5sec to be sure that the memory value is stabled.
return getAndroidDriver().getPerformanceData("appBundleID", "memoryinfo").get(1).get(5)) / 1024;
}This method also returns the information of the system state which is supported to read as CPU, memory, network traffic, and battery. In our case we only needed to get the memory value and as you can see we are dividing the value by 1024 to convert it to MB because this method returns the memory values in KB. Also, the threadSleep(5000) is for waiting 5 sec whenever we call this method just to be sure that the device has reached the stable state and called the Garbage Collector to deallocate all the unused references so that we can have the stable memory value.
Note: The .get(1).get(5) in the method above is used to filter the value that we specifically needed, which is the device’s TOTAL memory value, we needed to filter it as this method returns a list of lists and there are many other data which we don’t need in our case.
6. Reporting & Maintaining
We have created a Jenkins job to execute the tests regularly and whenever we want on any branch (Master, Release, Development). We have created a Slack channel and implemented the setup in our Jenkins job’s pipeline so that whenever we execute the job, the tests will run and after they finish, the failed tests will be reported to a dedicated Slack channel. Here’s what it looks like in Slack:
The first part is the Allure report link which gives more detailed information about the tests, but not everyone needs to deep-dive into the results. That’s why we have simplified the failed test results and printed them in Slack.
What is Allure Report? It is a flexible, lightweight multi-language test reporting tool.
You can see that it prints the test name and platform name (iOS/Android) and on which iteration the tests have failed, then reports the memory values for each iteration so we can see the increase of memory value gradually. Lastly, it shows the “Diff” value which is the difference between the latest iteration memory value minus the initial memory value(161.9–139.3=22.60). Therefore, as we have declared a 15 MB threshold for iOS tests, it printed the test that exceeded the 15 MB threshold.
7. Conclusion
By developing this memory monitoring tool and automating its usage, we have managed to prevent the most common memory leaks in our main flows of the application, decrease OOM (Out of Memory) crashes, and increase the crash-free rate of the application.
Thank you for your time!
Feel free to contact me on LinkedIn:
https://www.linkedin.com/in/nubar-nalbandian/
