What will storytelling look like in 20 years? Will it still be on your television? Will it printed on paper or projected in 3D? Prophesying the future is hard. But, like fortune telling with tea leaves, sometimes the future can be glimpsed in what’s here right now.
Last year, Charlie Brooker’s Black Mirror: Bandersnatch – a nihilistic choose-your-own-adventure style film with five main endings – introduced Netflix viewers to a term that has only recently entered the TV lexicon: interactive storytelling. Following up-and-coming developer Stefan as he works tirelessly to create the most complex video game of 1984, Bandersnatch calls on the viewer to make his choices. Do you angrily douse your computer in tea or yell at your dad to blow off steam? Do you visit a therapist or shirk the session to follow a mysterious colleague? Sugar Puffs or Frosties? Bandersnatch is an example of a growing trend in storytelling space: too interactive for traditional TV, not quite interactive enough to be a video game.
Since 2007, the Sundance Institute’s New Frontier programme has been an incubator for artists and technologists experimenting on the boundaries of narrative, from tales told with haptic feedback to storyworlds explored in digital simulations. Now in its 12th year and in partnership with video game technology company Unity, New Frontier 2019 is a showcase of virtual, augmented, and mixed-reality storytelling, experiences that exist in an in-between space: not quite video game, not quite film, not quite theatre.
One exhibitor is Melissa Painter, who explores the tech-art intersection in her work. Her MAP Design Lab created MoveStudio, an exploratory experience for Microsoft VR headsets that allows players to manipulate the world around them, and is also behind augmented reality projects such as Heroes for HoloLens. It’s a leap she made six years ago after a successful career as an independent film director: Painter first entered Sundance in 1993 with her graduate thesis, a film called Jump.
But after years in the industry, directing three feature films, followed by a move into documentaries, Painter turned to immersive storytelling. “I was a director,” she says, “a female director. Not necessarily an easy thing to be. And once I stepped toward emergent technologies I stepped away from film and never looked back. When I got my first taste of immersive storytelling I realised we were inventing a new relationship to technology that was going to alter not just culture, but the world, everything we do. It felt really essential to be part of that pioneering moment.”
Painter’s newest project, Embody, developed in partnership with the Canadian sports wear company Lululemon’s Innovation Lab, is a co-operative VR experience influenced by body practices from yoga to aikido. It starts with two strangers entering a small office. Behind the door is an even smaller white room bathed in a pale blue light – a kind of Kubrickian mudroom where my fellow stranger and I are asked to take off our shoes. We shuffle in our socks as we are led into the same room, but through different doors. My Embody-mate disappears behind a curtain and I sock-shuffle over to a mat and slide on a VR headset.
Embody puts me into a utopia with a 360-degree blue-sky world, surrounded by tropical flowers and the environment’s yogic hum. I’m guided through a number of movements by an on-screen companion – a translucent avatar who repeats a simple yogic pose until you mirror it.
The guide throws out its arms. I heave mine out too. The flowers breathe open as you stretch your arms out and back in the rhythm of the avatar. In a later scene, the guide ushers me into a tree pose in front of a large, virtual tree. It’s not the easiest move: try keeping your right leg on the ground, then press your left foot against the inside of your (hopefully) steady right calf. The longer you hold your balance, the faster the tree grows.
There’s no need to wear trackers or hold controllers. Embody figures out your movements from a camera, the headset and sensitive data from the mat under your feet. “Our goal was to take as much technology as possible off your body but to still have an experience that’s driven by the movement of the user,” Painter explains.
“We used machine learning, deep learning, machine vision, AI, everything, using a very simple 2D camera, and pose detection to capture how you’re moving. We wanted people to think about their own body, not the technology. I wanted to remind people that their body is a creative tool … I think a lot about the female audience. Gamer-style users get into my experiences and wonder what the hell I am doing. Women don’t ever seem to have a question about my intentions.”
While virtual reality can evoke the image of an action-packed Ready Player One universe, it has proved an immensely useful tool for mindfulness. Peaceful and isolated within the VR headset, it’s easy to forget about the world. When my avatar guide leads me to stretch out my right hand and press it against its translucent palm it’s startling to experience actual haptic feedback – the sweaty, human palm of the stranger I met earlier. It’s a fitting metaphor for immersive storytelling – something that literally touches the divide between the digital and real.
Gloomy Eyes, another project on show at New Frontier, is a rather less relaxing interactive experience. It’s set in 1983. The dead have risen and live as an illegal presence in a wood on the outskirts of a town. But this is also a love story. On a cold night in Woodland City, accompanied by the soft sounds of Erik Satie’s Trois Gymnopédies, a mortal girl called Nena meets Gloomy, a lonely zombie boy. Gloomy Eyes presents a story as a physical space, entertainment that can be explored virtually with your body – an intriguing new prospect that VR technology has made possible.
You dictate the pacing of the scenes from inside the pillowy darkness of a VR headset, kicking off when you turn your gaze toward a light just over your shoulder. Woodland City appears like a living, floating diorama. Bend down to get a better look and the textures become larger, crisper, revealing a woolly-goth aesthetic comparable to that of Tim Burton’s Nightmare Before Christmas. The town comes alive with that same toy-like quaintness – lights flicker inside houses; a wiggly dog pants or yaps for attention.
“There was a time when the sun got tired of the idiocy of humans and decided to hide and never rise again,” says the narrator, a gravekeeper with insight into the worlds of both living and dead. The story is narrated by actor Colin Farrell, who joins a growing number of celebrities dabbling in virtual reality this year – New Frontier features projects by experimental comedian Reggie Watts (Runnin) and android-like avatar-turned-pop-singer Poppy (The Jester’s Tale).
Events unfold around your body. To my left is the small town that houses Nena and a group of hunters who patrol the area for zombie threats. A light guides my gaze to the right, a 3D woodland set. Further right to is a swampland and a broken bridge where Gloomy lives. As Nena and Gloomy greet each other for the first time at the bridge, pieces of wood detritus float up as if by magic and restore the structure to new.
By the end of each story beat, the light returns to guide my eyes to a new set floating nearby in the blackness. I shuffle around these worlds as if they are infinitely delicate train model sets and I am a fat-fingered toddler. Once I lift off the headset, I’ve turned around so many times that for a moment I can’t work out which way I entered the room.
“We want to do things that look very theatrical, very small,” says Jorge Tereso, Gloomy Eyes project director. Something that happens in VR as a director is that you lose control because you don’t have the composition of the frame. By scaling everything down we can get that back.
“There was a lot of trial and error because there are no written rules yet in VR,” Tereso adds. “You just have to see what works. Like trying to invent ways to do cuts, to make it clear that time had passed.”
From cave paintings to radio broadcasts to VR, storytelling is a way of making sense of the present. For New Frontier’s director Kamal Sinclair, this is particularly the case in times of disruption and change. “Mass media has changed our sense of reality,” she says. “We saw Earth from space and it changed our sense of what real is. We saw Vietnam from our living rooms and saw the frontlines of war. That changed how we had to perceive reality. Artists stepped up to make meaning out of that disruption.”
“We’re at that point again,” she says. “And the artists are showing up again.”
Comments (…)
Sign in or create your Guardian account to join the discussion