The iPad as a fast, precise tool for creativity
Science fiction suggests that tablets are the computers of the future. From Star Trek to Westworld, a clipboard-shaped computer (often with stylus) seems more natural for humans than a desktop computer with keyboard and mouse.
In 2018, the iPad Pro is the most sophisticated tablet to date. How close is it to matching a Mac, Windows, or Linux computer for creative or productive tasks?
Not very close, it seems. Our user research indicates that most creative professionals — that is, artists, scientists, writers, designers, software engineers — mainly use a desktop or a laptop computer for their work. They often like the idea of tablets, but in practice their iPad is used for Netflix and social media on the sofa. As application developers, we might ask why.
One possibility is that tablet apps are mostly designed as scaled-up smartphone apps. And phone apps are designed for consumers: optimized for zero learning curve rather than power or flexibility.
Watch a creative professional using a tool like Photoshop, AutoCAD, vim, or Excel and see how quickly and confidently they can make complex changes to a document. What would it take to create a fast and precise tool like this on a tablet?
The Ink & Switch research team set out to explore this question. Our preliminary findings are that a tablet can be a powerful tool for creative tasks — but only by challenging some of the design orthodoxies of mobile app development.
The prototype
Our proof-of-concept is an iPad app called Dossier.
Dossier is Evernote meets Pinterest. You can capture text, images, links, PDFs, and any other file format into a collection. Each item displays as a card which can be arranged spatially on a virtual pinboard.
Dossier requires a stylus and uses Dropbox for storage.
Challenging the mobile app design orthodoxy
We built this prototype with five unusual interface ideas in mind, as follows.
1. Stylus required
When I got started with computers running MS-DOS in the 1980s, the mouse was an optional accessory. Software developers couldn’t assume it was available. Today, the stylus gets the same treatment. Because most tablets (such as the iPad or Microsoft’s Surface) don’t come with a stylus by default, app developers treat the stylus as just another finger, albeit a more precise one.
For Dossier, the stylus is required, opening new possibilities for command richness.
2. Put your hands all over it
I find tablets appealing because they are similar to my A5 paper notebook. It’s a delight to sketch in my notebook at a coffee shop, at my desk, or sitting on a park bench. A tablet is roughly the same size and shape, but often doesn’t offer the same joy of use.
One reason is that tablet apps typically have many active buttons and controls on the edge of the screen. Given the nearly edge-to-edge screen, it’s hard to grip the device without accidentally pushing a button or scrolling the page. I end up holding my tablet with the tips of my fingers, vaguely afraid to grip it the way I would my paper notebook.
Apple’s Keynote has active controls cover two out of four edges of the screen
For Dossier, we tried to have a minimum of active controls onscreen, particularly near the edges.
3. No-wait commands
Touchscreens have a small number of gestures: tap, swipe, double-tap, pinch, and a few others. Smartphone apps further assume a small screen and one-handed operation — which means many commands end up tucked away behind long-press or other commands with built-in delay.
Dragging items between pages is similarly friction-inducing: users have been trained to drag an item to the edge of the page, then wait a while for the scrolling to kick in.
Perhaps the worst offender for power users is copy and paste. On the desktop these crucial commands can be invoked almost anywhere with ctrl-C or ctrl-V. On mobile, it’s a fussy process of accessing a context menu.
For Dossier, we made it a goal that no input method would require the user to wait, even briefly.
4. Read the manual
Consumer app development assumes that users must immediately understand how to use the app with little to no instruction.
But creative professionals are different. We are okay with investing some time learning a tool if that investment pays off in power, flexibility, and expressiveness using the tool later on. Again think of packages like Photoshop, vim, AutoCAD, and Excel.
Further yet: we appreciate documentation. A product that comes with a manual implies it has depth.
For Dossier, we ask the user to read the manual, considering that a reasonable tradeoff for a more capable tool.
5. Command vocabulary
Every user interface has a command vocabulary. Desktop platforms typically show their full vocabulary including keyboard shortcuts in the menu bar. Combined with a pointing device and modifier keys and you have thousands of discrete inputs that can form the app’s command vocabulary.
On a tablet, the usual list of input gestures (tap, two-finger tap, swipe, pinch, etc) is quite small. Even allowing for two hands (e.g. pinch with one hand, tap with the other) and our potential command vocabulary only counts in the dozens. But the stylus opens up a whole new class of input: writing.
Our team took inspiration from the 1991 tablet operating system PenPoint OS. Among it many gestures, PenPoint allowed the user to draw glyphs over the top of onscreen objects to issue commands. For example, a circle to edit the item, and an X to delete it.
For Dossier, we resolved to build a rich command vocabulary using stylus gylphs in addition to all of the finger gestures.
Putting it all together
Using these five premises, we built the prototype app as follows:
- Stylus required: We take advantage of everything at the disposal of the average human: two hands (including ten individual fingers) and the stylus as distinct input methods, sometimes used in tandem.
- Put your hands all over it: Dossier has almost zero chrome, allowing the user’s content to occupy the entire screen, and very few buttons activated by a single tap.
- No-wait commands: Nothing in the Dossier command vocabulary requires long-press or other delay. The common operation of moving a card via one-finger drag responds instantly, metaphorically like sliding index cards around on a table.
- Read the manual: Dossier has a cheatsheet available in the main menu which describes the full palette of commands available to the user.
All of this comes together with point 5, the command vocabulary. Commands such as copy, paste, and delete (normally hidden behind long-press context menus on mobile applications) are available by drawing a glyph with your stylus. We recognize glyphs using the $1 Unistroke recognizer as implemented in Swift.
Each command gives a flash notice (inspired by Pixelmator on Mac) that matches the command name shown on the cheatsheet. Draw an X to delete and you see “delete card” as well as the card disappearing animation. Draw that same X over a page in the collection overview and you see “delete page.”
Also included is the more common use of the stylus: freeform sketching. In Dossier it takes the form of annotations with a feel similar to drawing on a whiteboard with a black marker.
See the full tour in this 2-minute demo video:
Findings
Our team tested the app internally for our own use cases: project brainstorming, travel planning, meal prep notes, an office remodel project. We found it a useful alternative to Evernote, Pinterest, or Google Docs for this type of “get your thoughts together” or “make a plan” work.
We tested Dossier with four external users. In each case, the user exhibited initial confusion that mobile app conventions such as one-finger swipe didn’t work as expected. But as they consulted the cheatsheet and experimented with the commands, they developed an ability to work with the tool within a few minutes. Our takeaway was that the learning curve was surprisingly small relative to the payout in capabilities offered by Dossier relative to other iPad apps.
In my personal experience using the app, I really enjoyed quickly pulling in text, image, and PDF cards from different sources and grouping them spatially to find patterns. I also found Dossier useful as sort of fast, low-effort slide deck creator for ad-hoc presentations over screenshare with my team.
Conclusion
We set out to question some common mobile design practices in pursuit of making the tablet a fast and precise tool. The result was the command vocabulary with no-wait commands, stylus glyphs, a flash notice to reinforce the user’s actions, and a mini-manual in the form of the cheatsheet.
Based on these preliminary findings our team believes that the tablet can eventually be a viable platform for creative professionals. But to achieve that may require app developers to break free of some assumptions from the consumption computing world.
We’re sharing our findings in hopes that other digital toolmakers might borrow these ideas for their production apps, or that other researchers might continue to explore further in this direction. Further ideas to explore include multistroke recognizers, voice commands, and chorded keyboards.
We’d love to hear what you think or hear about your work in this domain. You can reach us on Twitter or at hello@inkandswitch.com.
Update August 2019: Want to try it? Many of the ideas from the Dossier project will appear in Muse for iPad.
Dossier was built by Julia Roggatz (engineering), Miloš Milikić (design), and me, Adam Wiggins (project lead). Special thanks to James Lindenbaum, Roshan Choxi, Orion Henry, Mark McGranaghan, and Martin Kleppmann.
