Filmmakers are using AR and Apple’s ARKit to create extravagant short films

ARKit, Apple’s augmented reality development kit, has arrived with the release of iOS 11. While the focus of what developers can do with AR has been primarily on games and helping with day-to-day chores, there’s another interesting sector of entertainment developers are using AR to create: film.

Duncan Walker, an independent game developer based in London, England, used Apple’s ARKit and Unity 3D to make a short film in his iPhone 7. The film took a weekend to create and edit — something that would have taken weeks, if not longer, to do prior. The visual effect of the soldiers marching through the town are 3D rendered and, most importantly, done entirely on his phone. Moving the iPhone around his neighborhood and using the scenery involved, Walker is able to add virtual effects to the real world setting and produce a short narrative film.

“When ARKit came out, I got it and put some characters I already made in there and thought, ‘Well, these look good enough so let’s try and make a little film out of it,” Walker told Polygon from his home in London.

Duncan Walker

The effect is a project that Walker calls “AR Film Experiment.” It’s an ongoing, evolving experiment to see just how far Walker can use ARKit to create short films. The final project, Walker told Polygon, will be a short film that he presents at a festival. Since debuting the first iteration of the project on YouTube last month, Walker said there has been mixed reaction to what he’s trying to accomplish with ARKit. There are enthusiasts and newcomers to the scene who marvel at the unique application Walker has come up with for AR and then there are those who work in the film industry.

“There are these people who don’t know much about it but think it’s really cool and exciting and then VFX and film people who tell me that the lighting is incorrect,” Walker said laughing. “You know, I know all of these things! But both groups understand what I’m trying to do with the project and they dig it.”

When Apple first announced its ARKit, a platform for developers and iPhone users to share and download various AR apps, the focus was on home renovation (like the ruler app seen below), day-to-day use and, of course, games.

Pokémon Go’s success proved to Apple and the world that there was a hunger for cool AR experiences people could carry around in their pockets. Games are still very much considered the ultimate frontier of what AR can accomplish, but AR isn’t only for games.

The television and film industries have used AR technology, alongside virtual reality (VR) technology for years. MLB Now, a one-hour panel discussion program about major league baseball was given an AR overhaul by BigStudios, a design firm in Toronto. As part of the concept, BigStudios imagined how AR could be used without making the viewer aware that there was an additional layer of technology. Simply put, it had to look good and authentic, not just flashy. In an example, seen below, players that were being profiled stepped out of their boxes and onto the stage.

With ARKit and ARCore, Google’s software development kit that will allow developers to bring AR capabilities to Android devices, the potential for projects to use AR is endless. The biggest change, Walker said, is instead of working with major design studios on big-budget programs, anyone with the know how and capability to work with AR can contribute their own film. Apple and Google’s ARKit and ARCore will allow AR experiences, games and movies to be distributed amongst millions of iPhone and Android users. If the App Store forever changed how games were created and shared on mobile when it first debuted in July 2008, ARKit has the potential to do the same for AR in 2017.

“What I’m trying to prove with the experiment is that this type of filmmaking can be open to anyone, not just the big studios,” Walker said. “Independent filmmakers, game developers or anyone really.”

That’s not to say Walker hasn’t faced his fair share of problems. The developer considers himself a grassroot filmmaker, to an extent, going into the world of AR and short films without any guidance. ARKit and ARCore mark the first time that independent iPhone and Android users will be able to mess around with AR and share it with others. Without a studio backing, Walker admitted it can be difficult to find answers to a question through a quick Google search.

Especially considering that at the time of the interview, iOS 11, which is equipped with ARKit, wasn’t available to the general public just yet.

Walker’s problem isn’t exclusive to him, though. There are dozens of developers and filmmakers who want to play around with new technology but don’t know how to begin. Like any new industry or trend, however, there are people circling, waiting to get in on the action and lend their assistance where possible.

Duncan Walker

Philippe Lewicki and Jesse Vander Does are the co-founders of AfterNow, a development agency that that specializes in AR and VR. AfterNow has worked with Disney, Marvel and Warner Bros. on numerous projects and, with the release of ARKit, is excited to see the new batch of talent that comes forward.

AfterNow has a focus on integrating AR and VR into major studio projects, but in their time helping bigger companies, Lewicki and Vander Does realized the potential was there for independent filmmakers to use AR to their advantage. One of the most applicable ways AR can be used for projects both big and small is in the planning stage, Vander Does said.

Picture this: You need to visualize how an actor may look while jumping across a canyon, falling through the air and landing in the river below. Instead of relying on storyboards or spending time and money on resources to create a visual representation, a filmmaker can hold their phone up and create the scene, getting a better understanding of what they can expect in the final product.

“When you’re trying to visualize things, getting the best image possible is key,” Vander Does told Polygon. “People are really used to looking through a director’s viewfinders and seeing what’s in front of them. What AR on your phone can let you do is present what the framing will look like. ‘How will this look if I put a five-foot actor over in that corner underneath this sign?’ Framing is key and I think that’s sort of where we are right now with how AR can be used practically. We see this evolving into the actual shot pretty quickly.

“Eventually you’re going to be able to make an entire movie in an app using AR, there’s no question about it.”

There are a few other apps that have been developed specifically for ARKit to help people make their own movies. One app, AR Movie Maker, works like Snapchat, allowing people to scroll through a selection of characters, implement them into the area their phone is pointed at and assign those characters actions. Once everything is in place, a narrative can be threaded using the app’s story building tools.

Another app, Studio Director, gives users various tools — including 3D rendered actors, props and different lighting settings — to make movies.

Although these apps were specifically made with ARKit, Apple’s development platform isn’t the first example of a company trying to bring more AR experiences to mobile. Google launched Tango (known then as Project Tango) in June 2014. Tango allowed developers to create AR and VR experiences for a few different smartphone and phablets. The impracticality of the size of phablets that work with Tango, however, have impacted how many people are actually using Tango for AR purposes.

As The Verge pointed out when ARKit was first announced:

Apple's AR will immediately reach millions of people who already have the requisite hardware. And while it looks to be functionally as flexible and capable as Google's Tango (check out some early examples of fanciful experiments with ARKit), its broader audience makes it much more enticing for serious developers to invest their time and money into. Google's Tango is about the future whereas Apple's ARKit is about the present.

Vander Does agrees. The developer, who worked for years as a visual effects artist in the film industry, pointed out that AR will go from “Is this something that people are really going to want to have on their phones?” to “How quickly can we get our product into people’s hands?”

“When ARKit landed, suddenly the prospect was a bigger sell for developers and filmmakers than ever before,” Vander Does said. “Millions of you already have this in your pocket. It’s not as high quality as Tango, but that doesn’t matter because it’s outweighed by the reach. ARCore is going to be relatively easy to port to Android and Google will be able to give developers and filmmakers even more of an opportunity to deliver their product.”

The most exciting aspect of AR development and a focus on film for Walker is giving people the opportunity to create the world they’ve always wanted to exist in. Walker believes this is just the beginning of what AR is capable of and wants to move toward being able to create features like Ghost in the Shell on a phone. For now, being able to hold a phone up and point it around your surroundings to watch a short film play out on your iPhone or Android device is still pretty exciting for the developer.

“We’re adding virtual stuff to footage and using the real world as a way to watch it all come together,” Walker said. “We’re not there yet for how I imagine AR will look in the future, but we’re finally bringing our imaginations to life with nothing but a phone and little bit of technical skill. That’s really, really cool.”

iOS 11 is available to download right now — with full access to ARKit and the various apps already being developed.