Continuous Quality: Ship Better Apps Faster with Visual Studio App Center
Think about one of your mobile apps… How often do you publish updates to it? A couple of times per year? Quarterly? Monthly, bi-weekly? Every day? Multiple times per day?
No! That’s crazy, right?
Or is it? Some world-class app engineering teams are able to consistently release their apps weekly or every two weeks at high quality. How do they manage to do that? Part of the explanation is commitment to a continuous delivery process in some form. Let’s look at some of the benefits and challenges of continuous delivery, and, more importantly, why this is accessible to everyone. You don’t need to be a venture-backed startup or a huge company to do this stuff!
Benefits of Continuous Delivery for Mobile Apps
A continuous delivery process entails being able to release quickly, with high quality, at any given time. Let’s contrast this with what we’ve seen a couple of times. The business wants a critical hotfix pushed out ASAP?
- NO! Fred is on vacation and he’s the only guy on the team who can build, sign for distribution, and upload to iTunes.
- NO! We’ve not gone through the full QA cycle for the current master branch, and it will take three days to go through the manual regression tests, and, by the way, who’s got the iPhone 4S running iOS 9.2?
- NO! We’re not entirely sure how long the release process takes; reviews could take anything from 5-12 days.
Sound familiar?
With a continuous delivery process, things become easier. The business wants a critical hotfix pushed out ASAP? Not a problem! A developer makes a branch/pull-request with the change and gets peer review. An automated process builds and regression-tests the app using a CI server, test lab, and battery of automated regression tests. The developer merges the change to the master branch and pushes the release button! Continuous Delivery reduces your lead time, getting changes from idea to end-users faster. It reduces risk and variance in the release process because of automation and high repetition.
“Sure,” you might say, “but we don’t have the resources to maintain such an infrastructure.” By the end of this post, you’ll hopefully see why you’re wrong; with Visual Studio App Center, this is accessible to everyone via the cloud at low cost.
Challenges for Continuous Delivery
App Store review times, a major barrier to continuous delivery in the past, have been declining steadily since 2015, and are now about one day on average. Still, there remain several challenges with implementing a continuous delivery process. For example, to build iOS apps, you need a Mac-based CI server, which is uncommon in most companies, and configuring this for CI can be time-consuming. With App Center, building in the cloud (even for iOS) takes less than five minutes to set up.
But what about quality? Thorough testing of apps is time-consuming and difficult. There are literally thousands of device models out there, with different and often limited hardware capabilities, screen sizes and resolutions. There are numerous platforms, including Android, iOS, macOS, and Windows, and your app likely supports multiple versions of each.
One way to address the challenge is to rely on outsourced manual testing with providers that offer specialist testers with access to large libraries of different mobile devices. This has benefits, but also drawbacks:
- You need to clearly communicate your requirements.
- You need to interpret the test results.
- The tests are performed by humans, who make mistakes.
- The turnaround for test passes can be slow, ranging from days to weeks.
- Integrating this testing process in to an automated Continuous Integration/Continuous Delivery pipeline can be hard.
Instead, what if you had access to a large set of cloud-hosted devices on which you could run automated regression tests of your app with a single line script in your CI system? What if the tests produced visual test reports, so that not only could you perform automated tests, but also be able to review layout and compare differences across device models and OS versions? That’s what Visual Studio App Center Test offers.
Getting Started: How to Extend Continuous Builds to Continuous Tests
It’s incredibly easy to add automated UI tests into your App Center builds. We have thousands of physical Apple and Android devices on which to run your newly-built app, and App Center supports the most popular UI testing frameworks: Espresso, XCUITest, Appium, and Xamarin.UITest.
To run a UI test:
- Using the App Center Test CLI, upload your test suite automatically after completing a build.
- Once you generate a test upload shell command, add it to your app’s repo at the solution or project level.
When your test code finishes executing, a detailed report will appear in the App Center Test and the CLI will return a URL linking directly to it. Please see this example command as a reference. Instrumenting your continuous builds with continuous testing is a sure way to shorten the feedback loop, greatly improving app quality.
When the test report is complete, you can easily identify the failed devices by the red “lightning” indicator. Additionally, you can filter just the failed devices or by other parameters by selecting the icon in the upper right corner of the test report.
A red “lightning” icon indicates a failed device.
For every test device, App Center provides device logs, test framework logs, and the line in which the failure occurred to help you easily identify the regression. Start by examining the test failures for clues about where in the test execution the failed assertion occurred. If this is not revealing, dive into the test and device logs for more thorough investigation to determine the root cause of the failure.
App Center provides device logs, test framework logs, and details about failures for every failed device.
New Features
Here are the top three features to try in App Center Test:
1. Named Device Sets
It’s best to run your app on as many devices as possible before shipping to your users. App Center offers named device sets to help you organize and reuse collections of mobile hardware in our labs. Try this out today by adding a “Top 20” set that you can update as you see trends in your users’ devices changing.
2. Launch Test
To ensure your latest app build runs and opens on our devices before you have any test code written, you can enable launch test in your build configuration. Validate your app’s ability to start on our wide variety of hardware today!
3. Improved API
App Center Test provides a robust REST API giving full access to all data generated during your test run. The data in your test report is accessible via our API, including screenshots, device logs, test runner logs, and failed device specifics. To get started with App Center API today, please see the following documentation.
What’s Next?
We’re constantly building new features, improving existing functionality, and better integrating with other App Center services to make testing even easier. In an upcoming blog post next month, we’ll deep dive into advanced tips and tricks for testing in App Center.
Log in or create your free App Center account to get started today!
Light
Dark
0 comments