App Developer Friends: Try TestFlight.
January 21, 2015 ∞
In March last year, after a particularly frustrating few hours dealing with iOS beta device slots, I filed this bug report with Apple. I didn’t realize it at the time, but a solution to this problem was already in motion. In February, Apple had acquired a company called TestFlight. Over the next few months it integrated many of the original TestFlight features into iTunes Connect.
Shortly after iOS 8 was released, Apple opened this new beta testing service to iOS developers. When compared to the previous testing process, it is a major improvement and I am grateful to the team behind it. It is a sign that Apple cares about third party developers and about helping us improve the quality of the software we provide.
TestFlight Features
- Developers can invite up to 1,000 external testers per app. The previous limit was 100 device slots for an entire developer account. Additionally, there was no way to recover slots by removing devices. All 100 slots could be reset only once per year.
- Beta releases are tied to users’ App Store accounts, not to their specific devices. Previously, builds were locked to a specific device ID. Every September, many testers would buy new devices and then need to be re-processed, using up another device slot.
- The TestFlight process is really, really simple for testers. All that is required is the user’s e-mail address to send them an invite. At Supertop, when we used Hockey for distribution, setting up a new user was a rigmarole. I wasted hours manually managing device IDs. By making life easier for testers, developers are likely to see much higher install rates. Testers will actually test.
- Test users can ‘buy’ IAPs with their real app store accounts. They are not actually charged, but the purchases remain valid for the period of the beta. This is another massive improvement over the previous system, where developers had to create sandbox test accounts and beg beta testers to sign out of their app store account (thereby deactivating iTunes Match), and then sign in with the sandbox one. In practise, nobody did it because it was such a pain. Now they can just use their normal account.
Notes
There are a number of issues to be aware of:
- Automatic crash reporting is a vital component of any beta test. Right now, TestFlight doesn’t do this at all. This is bananas. How does it not have this? For now, you still need to integrate your own solution to this. I highly recommend Hockey.
- After developers submit their builds to iTunes Connect, they can occasionally get stuck ‘processing’ for hours. It hasn’t happened to us, but our friends have experienced it and it can be very frustrating. Consultants who build apps for their clients could find themselves in a tight spot if, for example, they need a new build out immediately.
- You must get approval from Apple on the first build of each version of your beta. This has always come through in less than a day for us. Subsequent builds of the same version number don’t require approval (unless you’ve made huge changes AND you admit to it by voluntarily checking the box).
- Betas expire after 30 days. After that, the app ungracefully crashes on launch. Don’t let them expire.
None of these issues come close to negating the benefits of TestFlight for us as we work on updates Castro and Unread. We’ll be excited to beta test Castro 2 this way as well. App developers should take the time to try it. The availability of a much larger test audience, far better install rates and simplified IAP testing should help you release higher quality apps on the store.
Bonus Tip
Here’s a little tip I picked up from Paul Haddad on Twitter. If you want your app to detect whether it’s running on a TestFlight build, this should do the trick:
[[[[NSBundle mainBundle] appStoreReceiptURL] lastPathComponent] isEqualToString:@"sandboxReceipt”]