For years, futurists and filmmakers have lamented the passive experience of viewing entertainment enclosed within the frame of the screen. Even as devices become smaller, the rectangle still remains, delivering content to phones, tablets and computers. Over the next decade, emerging technologies promise to fundamentally transform content beyond the boundaries of the frame in unimaginable ways. Virtual Reality (VR) and Augmented Reality (AR) offer the most promising experiences. While VR is being popularized by video games and new immersive video content, AR also is poised to completely revolutionize entertainment on the road ahead.
You may recall that Oculus revitalized interest in VR back in 2012, and the virtual medium continues to shift the way we tell stories today. This year, consumers are beginning to understand the possibilities of virtual experiences as VR headsets come to the marketplace. VR and AR were on display at this year’s Consumer Electronics Show (CES), accounting for triple the amount of panels and exhibitors on the subject than in recent years.
The Sundance Film Festival also expanded its VR presence in January. Recognizing the crossroads of film, art and media technology, the festival celebrated the 10th anniversary of its New Frontier Program, becoming one of the first major film festivals to specifically focus on virtual story craft and new augmented technology.
In an effort to introduce VR to the masses, Facebook and Google recently rolled out 360 Video, a move designed to capitalize on the growing interest in immersive VR experiences. Additional platforms are now becoming available for VR content, while documentarians and filmmakers also embrace virtual technology to place their audiences within experiences to impact them in uniquely visceral ways.
Viewers of virtual content are given the feeling of being actively transported elsewhere, an experience commonly referred to as “presence,” an essential element of the VR experience. Yet for all virtual content, the true challenge lies in maintaining that elusive sense of presence for long periods.
Adam Orth, a gaming industry veteran and creative director at independent game studio Three One Zero, believes that basic “360 Video” falls short of living up to VR’s true potential. “The presence of VR only lasts a few minutes, at best, and you can’t do anything with it,” Orth explains. “This is where it falls flat. It feels incredible at first, but once that magic moment goes, the audience loses interest.”
Filmmakers such as James Cameron have also doubted VR’s longevity, feeling the medium is underwhelming, at best. Cameron has gone on the record as saying that VR equates to moving around within a video game—something he feels has already been with us for quite some time.
Cameron does make a valid point, as filmmakers will need to lace storytelling together with virtual interactivity in new and compelling ways to appeal to film audiences worldwide. However, there are many virtual possibilities coming to light in the video game space, projects even Cameron has yet to see.
Several VR projects recently launched rely on video game engines such as Unity, which allow the use of tracking and interactive elements. Oculus Rift’s interactive motion controllers also allow viewers to interface with content.
A new Oculus Rift game exemplifies the potential of VR. Developed by Orth at Three One Zero, ADR1FT tells the story of an astronaut stranded in space following a catastrophic incident destroying an orbital station. You awaken with no memory of the disastrous event, floating through the wrecked space station to learn about yourself, the crew and ultimately how to escape. As you travel, you encounter floating debris, personal effects and audio logs that reveal the story and allow you to make sense of your predicament.
This type of gaming is dubbed FPX (First Person Experience) and, when combined with VR, the experience is incredible. FPX places users directly into the main character’s POV to actively experience an emotional connection to story and place.
“I was not prepared for the hardware to tap into players’ real emotions the way it did while they were in the game,” notes Orth on the success of the game. “To a developer and creator, that’s like total gold!”
AUGMENT THAT REALITY
Despite VR’s promise, headsets present the practical dilemma of cutting off the user’s path of vision, enveloping your field of view. Clearly, this makes the headset inherently antisocial. While VR transports you somewhere else entirely by overriding your field of vision, AR is different. It enhances your view of the real world by overlaying it with information and holographic content. For this reason, content creation radically differs between VR and AR.
Microsoft has entered the AR space with HoloLens, an AR headset bringing high-definition holograms to life in its users’ worlds. During this year’s Super Bowl, Microsoft and the NFL partnered to bring what the future of football content might look like with a commercial for HoloLens, showcasing enhanced experiences where stats and information pop up all over the user’s field of vision. The ad also featured elements such as a 3D linebacker smashing out of a nearby wall, a 3D stadium of the game layered on top of a coffee table, and other elements and stats, making the case for a new way to experience live events such as the Super Bowl.
At TED 2016, a new AR headset company called Meta also blew away a packed audience of industry professionals while presenting their latest headset, the mind-bending Meta 2. Meta outlined a future for AR that reinvents how we interact with our computers. Their presentation even demonstrated a holographic phone call, with the ability to project a person into another room in full live 3D.
Like many startups, Meta was born from a Kickstarter campaign to fund development for one of the first consumer AR headsets. Despite the technology being in its infancy, Meta’s potential caught the eye of several backers, including Ryan Pamplin, a young Silicon Valley entrepreneur. Meta CEO Meron Gribetz kept in touch with Pamplin as development on the headset continued, work that piqued Pamplin’s interest so much that he left his own successful business to go work at Meta.
“We’re trying to change the computing paradigm,” explains Pamplin on Meta’s future. “We’re against the idea of a computer as an object that you lean over, build your desktop around and stare into all day long. We’re going to create a more natural experience. It’s not going to be a distraction like, for example, your phone notifying and informing you of stuff. Meta is about making the entire world your desktop background, and elegantly layering information over that world in a way that your brain wants it to be.”
Pamplin also says that holographic objects are being designed to behave in the same way as real-life objects. Interaction with such objects should theoretically be easy, and Meta continues developing software with neuroscience in mind. “We have this rule called the ‘neural-path of least resistance,’” Pamplin quips. “Things should work the way that your brain intuitively expects them to.”
Meanwhile, a prominent Hollywood director test-drove the Meta headset at TED, discovering its promise for filmmaking before production even rolls. He could see himself using the technology to collaboratively walk through a film set in AR mode to fine-tune things before building a single physical set. Other possibilities include the dynamic impact of using AR to interact with timelines and clips in 3D during postproduction.
“It’s this enhanced reality that makes life a lot easier,” adds Pamplin. “AR gets rid of a lot of interface compromises that have been made designing very complex menu options into basic devices. We’ve only ever worked in two dimensions, and now we’re adding a third, allowing interface with objects as if in the real world. It’s something your brain already knows how to handle, so there’s no learning curve at all.”
Meta’s TED demo also showcased full holograms of real people, something Pamplin explains can be captured quite easily. “This is something you can record with one camera to get a 180-degree view,” he says. “Or you can record with multiple, inexpensive cameras like the Intel RealSense 3D technology to achieve a full 360-degree view of that person.
“Imagine recording U2 in concert, then having the band play at your house party for everyone wearing the glasses, playing and jamming right there in front of you,” Pamplin says. “It looks real and sounds like they’re there. That kind of entertainment and the ability to tell new stories is endless. Imagine seeing a play reenacted at home. I’m talking about seeing life-sized actors standing before you. You’re not seeing a video in 3D, but real representations of people in your living room. That’s the difference between augmented reality and virtual reality. It’s about enhancing experiences, not just giving you an alternate experience.”
Between the immersive experiences of VR and the intimacy of AR, content creators are beginning to make deeply engaging art and entertainment in unimaginable ways. As developments continue and experiences improve, so, too, does our enhancement of the human condition, and the possibility to watch incredibly engaging storytelling shown in new perspectives beyond the rectangle like never before.
Visit ADR1FT at adr1ft.com, and Meta at metavision.com. Learn more about the HoloLens at microsoft.com/microsoft-hololens. Robert Powell hails from Chicago. An American Film Institute Conservatory graduate, he’s also a producer of film and VR content in Los Angeles.