Mobile App Performance Redux
Since my original post, I had requests for a more comprehensive test, so I decided to include more mobile development options. In addition to Java, Swift, C#, and Ruby, I ported my logic to Objective-C and JavaScript and C++. Objective-C allowed me to get a true baseline on iOS, and JavaScript allowed me to test webviews, mobile browsers and Titanium. Facebook’s React Native will get tested when it is released.
Testing setup
Just like before, I ran each test 10 times and took an average. However, I changed the logic a little bit from my previous test, which caused the new tests to be a little more computationally expensive. Here is all the relevant version numbers:
- Apple iPad Air 2 / iOS 8.2
- Apple iPhone 6 / iOS 8.2
- HTC Nexus 9 (2014) / Android 5.0.1
- Motorola Moto X (2014) / Android 5.0
- Xcode 6.3 beta 3(6D532l)
- Android Studio 1.1.0
- RoboVM 1.0 (1.0.0.201503101143)
- RubyMotion 3.6
- Xamarin Studio 5.8 (Xamarin.iOS 8.8.0.2 / Xamarin.Android 4.20.0.37)
- J2ObjC 0.9.6.1
- Titanium Studio 3.4.1.201410281727 / Titanium SDK 3.5.0
The source is located here: https://github.com/harrycheung/Mobile-App-Performance.
The Results
Here is the test results in spreadsheet form.
- Xamarin crushes my benchmark and is (no longer) the king of the hill. At over 30% quicker than Objective-C, I find it hard to find a reason not to use it. An eagle-eye will remember there is a big difference between this result and the result from my original post. @migueldeicaza pointed out I should not be using DateTime.UtcNow since it has performance limitations. It turns out I had some cruft so I took it out, and this was the result. The performance is so great that I have to imagine there is some magic involved.
- With the addition of Objective-C, we see that J2ObjC does a pretty good job of transpiling the Java code to Objective-C with only a 10% performance hit. When Swift is added to the picture, both the Objective-C and J2ObjC hybrid apps perform the same. This leads me to conclude the Swift bridge overhead was pretty significant.
- RoboVM has been officially released, and the performance is still outstanding, especially since I can compare it against Objective-C. I just really wished they supported Xcode storyboards.
- Of course, Safari and WKWebView exhibit about the same performance, and it is clear that Apple did some work to improve their JavaScript engine since both of those outperform Chrome and Titanium. The most telling observation from the JavaScript apps is how well they all performed relative to RubyMotion. RubyMotion is compiled down to ARM bytecode, and it still cannot compete.
- I find it a little weird that the Android WebView app perform so differently than Chrome on Android. I assume that both would have the same JavaScript engine, but it appears that they do not.
Update 3/17/15 9pm
@jl_hfl pointed out that I was handicapping Swift by using class instead of struct for Point. I restructured my code and we have a new king of the hill. I love seeing these performance increases with small tweaks. Please keep sending in the suggestions. Let us get each app optimized!
Update 3/19/15 3pm
Everyone has been asking for this so here it is. I ported my logic to C++. As expected, the C++ port performs the best on both platforms. What is incredible is how close Swift, Xamarin, and RoboVM are to C++ performance. Now, I will let someone else come up with an assembly port.
Update 3/22/15 5pm
I added benchmarks for a HTC Nexus 9 to give a perspective on how the best of both platforms compare. Note that Chrome’s JavaScript engine makes a big jump and is very fast on this tablet compared to the Moto X. There must be some major optimizations for the Nvidia’s K1 Tegra cpu on the Nexus 9.
Update 3/23/15 10am
I added results for the iPhone 6. When I ran these tests on the iPhone 6, I noticed that J2ObjC was outperforming Objective-C. I confirmed this on my iPad 2 Air. The biggest difference between when I originally ran the iPad tests and now is a little code restructure. Before, the code was not technically sharing the logic between apps. I copied the logic files to each project, and there might have been code divergence. Recently, I consolidated all the shared logic into one place so that each project pulls from the same files.
In general, it seems that the iPad Air 2 does not have a huge speed difference from the iPhone 6 in single core processing. The CPUs are very similar between the two devices with the exception that the iPad has one more core and a 100 MHz speed bump. However, notice that WKWebView performs the same on both devices. I am not sure what to make of that. Usually, that indicates being disk or network bound, but my app does not use any of those resources.
