Perfect Forwarding — Content app extensions in iOS 10

1.5M ratings
277k ratings

See, that’s what the app is perfect for.

Sounds perfect Wahhhh, I don’t wanna

Content app extensions in iOS 10

WWDC 2016 session “Advanced Notifications” packs a lot of material in showing how to create notification content extensions.  However putting together a working example required a little more digging, so here’s a summary of what you need to assemble.

Add the necessary keys to your notification payload

Two things need to be added to your normal notification payload to trigger an app content extension: the mutable-content key, and a category identifier to let the app know which notification content extension (with custom UI) to launch for the notification.

Optionally, you can specify images, audio, or video to embed in the custom UI for playback. These can be specified in the area outside of the aps dictionary in the payload in a format of your choosing.

image

Create a service app extension

The service extension is for processing content in the background before the user 3D touches on the notification to get the custom UI presented by the content extension.

First, create the extension through Apple’s wizard by adding a new target to the project your main app resides in.

Download the content and save to local storage

In the didReceiveNotificationRequest function stub presented by the wizard, add a body to process the attachments as you like. In my case, I’m downloading the image and going to stream the music, so I only need processing for the image. The general flow is:

  1. Set up a place to save the temporary files
  2. Download the attachments one at a time
  3. Create a new UNNotificationAttachment for each and add it to the modified UNNotificationContent
image

Some examples use NSURLSession to get the data and then save it, I’m using NSData’s helper function directly to get the data synchronously off the web. The service extension will take the file you’ve downloaded and move it into a sandboxed area for other processes (like the content extension) to access.

A note about memory and time

iOS places significant restrictions on the resources a service extension can use. The entire operation must complete in thirty seconds or less, otherwise the serviceExtensionTimeWillExpire method will be called and processing will have to end. It’s a good idea to minimize the size and complexity of content you want to move through the service extension. Memory also seems to be at a premium. Though I haven’t found any documentation provided guidance, I noticed my service extension was terminated mid-process due to memory constraints whenever I went over four megabytes of RAM in Xcode. Since Apple provides media players for audio and video, I’d recommend not even messing with those here and just grabbing the media URL from the payload in the content extension.

Create a notification content app extension

The notification content extension allows custom UI and media playback when the user initiates 3D touch on the notification. Create a content extension in the same manner as the service extension, but this time adding some specific data to its Info.plist.

  • category for the content extension
  • initial aspect ratio for the height of the custom UI (0.5 is a good place to start)
image

UI for the custom notification is just like any other, use Interface Builder to modify the default storyboard, and add outlets to your notification view controller as needed to modify content. When the user performs a 3D touch on the notification with a corresponding UNNotificationExtensionCategory, the extension will be launched and the view controller’s didReceiveNotification function will be called. Here you can process the newly assembled UNNotificationAttachment objects from yours service extension, and play streaming content.

To access file attachments iOS has moved from the the service extension, you need to request specific security access or attempts to open the file will fail.

image

Playing streaming audio is easy with the AVPlayer Apple provides. Since notification views do not respond to user input generally, there is a special exception for media playback, and Apple provides the UI for the control if you override a number of methods.

image

The delegate methods for media playback require an explicit frame for the button to appear in. This unfortunately is a bit antiquated and not in line with the spirit of autolayout and storyboards. The time that this function is requested by iOS is before layout may have finished in your views, so referencing other elements in the storyboard for their frames may not be appropriate.

The mediaPlay and mediaPause functions are called when the user toggles input on the system media player button in the UI. In this case, just hooking them up to the AVPlayer functions is sufficient.

How to debug an app extension

This took me a little bit to figure out, but since extensions aren’t really apps themselves you can’t run them directly from Xcode. First, set your breakpoints in your extension code, then launch an extension with the run button in Xcode. Xcode will then ask for a hosting app to start the debug session, it doesn’t seem to matter which. I’ve had success with Compass and Today (system apps). 

image

Then when the notification is received by iOS, you should hit the breakpoints first in your service extension, and then after a 3D touch, your content extension. At this point you’ll have access to all of the notification, its body, payload, etc. in order to process and debug your data.

Open issues

  • I’ve noticed that leaving the phone on a notification with custom UI maximized changes over time. Parts of the view disappear, strings return to default values, etc. 
  • A number of cryptic messages about async failures appear in my log window. 

I’m not sure if these are anything to worry about yet.

image

All of these may be due to the beta nature of Xcode 8 / iOS 10, or I could be doing something wrong. Have any new insight? Was something unclear? Let me know. :)

References

ios10 xcode 8 beta notifications app extensions ios 10 beta wwdc 2016

See more posts like this on Tumblr

#ios10 #xcode 8 beta #notifications #app extensions #ios 10 beta #wwdc 2016

More you might like

When app extensions don’t launch

I had a difficult bug to root out regarding app extensions (related to my previous post).  One of my apps could launch the notification service and content extensions without fail, whereas another would only show old-style iOS 9 notifications.

Looking at the device log I noticed when a notification arrived, iOS could not find an appropriate extension to launch, despite the “mutable-content” entry being set properly in the payload.

SpringBoard(UserNotificationsServer)[57] <Info>: Received incoming message on topic se.mydomain.myapp
pkd(MobileCoreServices)[88] <Notice>: No plugins found to match query <private>
pkd[88] <Info>: match returned 0 plug-ins

Investigating my successful case, I observed that the .app container housed a plugins folder which included two .appex binaries (for the service and content notifications).  In my app which wasn’t working, however, the same extensions had the file extension .app (as opposed to .appex). 

Checking the Packaging series of Build Settings in Xcode for the extensions, I noticed a .xcconfig file was overriding the filename for the extensions, and accordingly causing iOS to not find any .appex that could service the notification.  Correcting this build setting fixed the problem and notifications were thereafter parsed properly.

ios10 notification-extensions xcode app-extensions xcode 8 beta xcode8

C enum types in llvm 8.0

I have some old third-party (1990s) C code in one of my iOS projects.  Occasionally updating to a new version of Xcode breaks the build with a lot of the dodgier elements left in this code.  Updating to Xcode 8.1 resulted in some multiply defined symbols in code that previously linked without trouble.

SomeHeader.h

 enum {
	ST_LOCKINT = (1 << 0),
	ST_DD = (1 << 1),	/* ix */
	ST_FD = (1 << 2),	/* iy */
	ST_CB = (1 << 3)	/* DDCB FDCB (ii+d) write back */
} KMZ80_STATE;

vs.

enum KMZ80_STATE {
  ST_LOCKINT = (1 << 0),
  ST_DD = (1 << 1),	/* ix */
  ST_FD = (1 << 2),	/* iy */
  ST_CB = (1 << 3)	/* DDCB FDCB (ii+d) write back */
};

Functionally for the client code these two implementations are identical.  However the first means of defining the enum actually defines an unnamed enum and an instance of it _KMZ80_STATE.  This is a no go if multiple C files include this header.

llvm Xcode8 enum

Debugging Xcode app terminations from NSFileHandle

I had a situation when initializing my app where large sets (>1400) of files being cataloged were causing the app to quit immediately, with no formal crash log or exception triggered in the debugger.

Terminated due to unknown Memory condition

read the dialog box, and a Google search didn’t turn up much.  Looking at the incomplete crash log in XCode, the inciting process was listed as “unknown”, but several of the active processes had “[vnode-limit]” written in the “reason” column of of active processes.  I noticed also that the “fds” read an even 1600.

In my async startup code which read files out the Documents folder and added entries to an sqlite database tagging their metadata, I had a loop that went something like this:

for all files
{
    if ( file not in db )
    { 
        NSFileHandle* file = loadFile:filePath;
        // ... do stuff with file, add to db
    } 
}

I thought the scope of the if block would have the file handles purged and closed, but this doesn’t seem to be the case.  Adding a [file closeFile] to the end of the if block remedied the random crashes.

Unknowningly I suppose I hit a limit of consecutive open files, though the debugger didn’t make this quite clear.  If you run into a similar abrupt termination to your app, check how many file handles you have open.

ios xcode crash file-management

GaMBL Alpha Released

I’m currently working on an open-source game music player called GaMBL (Game Music Box for Lion), which has reached an alpha state and I’m looking for testers and feedback.  

Supported systems include 8 and 16-bit Nintendo and Sega systems (NSF/GYM/SPC/etc.). 

僕が開発してるOS Xゲーム機音源エミュレータがα版に辿り着いた。ファミコン類、ゲームボイ、メガドライブなど対応しています。今テスターのバランティアーを募集していますが、誰か興味があれば連絡してください。

You can grab the binary and documentation from the Github project page, please open issues there for any problems you have.  

https://github.com/dgventura/gambl/wiki

gambl open source game emulation game music osx

Perforce client path error resolution

Last week when running my Jenkins master/slave locally I ran into some Perforce issues that were breaking my builds.  While trying to sync I got an error message along the lines of

error: File
/Users/david/.jenkins/jobs/test/workspace/Projects/blah/lib/libFlurryAnalytics.a 
is not inside permitted filesystem path 
/Users/david/Perforce/perforce_1666/david_personal

“is not inside permitted filesystem path” is the error, and I couldn’t find this string easily in a number of Google searches, so I asked Perforce support for help.

The issue at hand is that if you have P4CLIENTPATH defined in the scope of your Perforce process, basically any commands to grab files (sync) outside of that scope will produce the error above.  This happened for me because I added some new workspaces for different Jenkins jobs, and they lay outside of my original Perforce home folder.

This environment variable can be set through the shell of your using, or via the P4CONFIG file.  In my case it was defined in my .tcshrc which ran every time I opened a terminal session.  

You can remove the definition of the variable, or undefine it before you run Jenkins to resolve the issue.

More info about the variable can be found on the Perforce website, though the particular error string it can prompt is not (hence my blog entry to hopefully aid someone else with a similar problem).

http://www.perforce.com/perforce/doc.current/manuals/cmdref/env.P4CLIENTPATH.html

cvs perforce jenkins environmentvariables

Automating VPN for Jenkins SCM

Making Jenkins builds effortless is quite easy with the build steps and depth of plugins available.  I’ve been working the nomad lifestyle for my employer the last several months, so VPN connection before source control management is a given.  When coding which produces the resultant continuous integration builds this is already taken care of, however my nightly builds would usually fail as the machine wasn’t on the company’s network at the time.  So Perforce fails to connect and the build tanks.

Fortunately, using a Jenkins plugin and a little scripting this is easily remedied.

Steps:

  1. Get the pre-scm-buildstep plugin and add it to Jenkins
  2. Write a script/batch of your choice to connect to the VPN
  3. Add a pre-build step to run your script per the plugin documentation
  4. Enable the Jenkins job SCM retry count if your script spawns without waiting for a return value.
  5. (Optionally) Add a post-build step to disconnect from the VPN.

In my case, I’m running Jenkins on my MacBook, so I use an AppleScript to connect to the VPN, and then osascript to run it.  Here’s an example of an AppleScript to connect to the VPN.

image

There’s probably a more sophisticated way to handle VPN connection failures and report them, but this does the trick right now.

jenkins vpn scm applescript continuous integration

Professionalism in code maintenance

Several weeks ago, a fellow I went to school with was decrying bad spaghetti code, sending out a call to trash the entire project and start over.  I reminded him the age-old aphorism that the majority of software cost is spent on maintenance, which after not finding in me a shoulder to cry on, he irritably disregarded.

As engineers we dream of building something fantastic: a tower, a tribute to the heavens that will be elegant in form and function, something our peers can marvel at.  But as powerful as our favorite language’s paradigm is, it’s nothing but vain ego-stroking unless it gets the job done.  I’m not devaluing clean, fresh code.  On the contrary, every time we add another inspector to a class, for the sake of maintainability the ramifications need to be thought through.  But the truth is most every kind of module you need already exists, and much of it is in a licensing model compatible with what you’re working on.  The best code you can write is the code that you don’t.

Fact being money-wise you’re going to spend most of your resources maintaining something written long before, so the least you can do is muster a positive attitude and see it as a chance to bring more value with less effort.  Reading, understanding, and utilizing other developers’ source (which by the way is pretty much the same as anything you’ve written and moved on from more than six months ago):  these are some of the most important skills you can refine as a software engineer.  Knowing what the code was originally intended for, and understanding the thought behind its creation will pay off many times over.  Fail to heed this and along the way you’ll find yourself rewriting a lot of modifications.  The other side of the coin is unless you’re working on a throwaway prototype for a personal project, it pays to spend some time documenting your code and make it easier for other developers to reuse down the line.

Recently I started work porting a Carbon-based frontend of the popular game console emulation library GME to my native 64-bit OS X Lion.  In about eighty hours of work I got from a zip file of code that didn’t compile to an app rewritten in Cocoa and Core Audio that replicated the core functionality.  I did this in less than 500 lines of code switching out only the GUI, application timing, and audio interface, while maintaining the essential application flow.  This was possible because the source was clear, consistent, and the key points were well documented.  I didn’t agree with a lot of the mechanisms and models used, but I understood the motivation behind their selection and respected the unity of the entire package.

The key challenge for my project moving forward will not be restoring the stubbed-out functionality or modernizing the UI, it’ll be preserving the code that works intact, and clearly demarcating my modifications and extensions.  Because in another ten years when I’ve long since stopped working on the project and some other developer wants to cut their teeth on programming for the latest platform’s APIs, I owe them the same professionalism the original developer showed me.

Responsible coding is accepting the realities of software maintenance and doing your part for the next generation.  The more attractive we can make the the implementations that exist, the less chance someone will waste time reinventing the wheel.

maintenance softwarereuse documentation
See this in the app Show more