Exclusives

By Mario M. Kenyon, Francine Marchetti, Furious M –

Virtual reality (VR), is the NEW big thing for narrative storytellers and production studios. For live action and animation, the process of creating quality VR starts with the story — which impacts almost every single part of production, post and distribution. With so much money being invested into this medium, it’s important to be educated and understand the many different formats that fall under the blanket term VR. These formats range from live action 360º video to computer generated (CG), real-time rendered experiences. The latter format allows for full immersion of multiple senses and interactivity that is commonly referred to as true VR.

So what’s the difference between 360º video, true VR, AR and their hybrids? To get a sense of how these different mediums compare and contrast we need to consider how each format is created and also how it is actually consumed by the viewer or user.

The manner in which these different formats are consumed is very important and dictates the level of immersion. For example, mobile VR head mounted displays (HMD) like the Samsung Gear and Google’s Daydream do not currently allow for positional tracking, which means the user can’t physically move around in the virtual space. They can only look around from a fixed position. Desktop HMD like the Oculus Rift and the HTC Vive have positional tracking so they can precisely track where the user is sitting, standing or moving in the virtual space. These differences determine how a story is told, including the level of interactivity.

Virtual Reality: VR is categorized as an immersive multimedia or computer-simulated reality that replicates an environment that simulates a physical presence in the real world, an imagined world, or a hybrid of both. This allows the user to interact in that world. VR simulations artificially create sensory experiences that can include sight, touch, hearing, and even smell. There are actually headsets out there today that release scented odors that drive an olfactory response during the experience, as well as wearable haptic devices that recreate the sense of touch to the user through forces and vibrations.

360° Video

360° Video or Spherical Video employs video recordings of real world content as the core media. The real world content is recorded using a cluster of cameras pointed in all directions at the same time. The resulting footage is then stitched to form a single video. This stitching process is either done by the camera or through specialized software that analyzes common visuals to synchronize and link the different camera feeds together. With 360º video, the viewer has control of the viewing direction in playback but can’t move in the space. The vast majority of live action 360º video today doesn’t allow for positional tracking as a result of the camera. There is simply too much data to capture. However, Lytro will soon be coming out with a 360º camera that captures enough data that it allows for positional tracking of the users head when playing out on an appropriate HMD.

Confusion surrounding 360º videos is common. The debate typically centers on whether 360º videos should be categorized as VR at all. Much of the issue stems from the fact that 360º can be watched on a wide variety of devices, further complicating the meaning of VR. For instance: watching a 360º video on your computer or on a mobile device without a headset does not qualify as a VR experience. In that case, the user is not immersed in a simulated environment. If the 360º video is consumed using a mobile or desktop headset, it can be considered VR, although achieving presence is much more difficult due to the lack of positional tracking.

This highlights the importance of creating high-quality or “AAA” storytelling, and why creating well-crafted stories is so important in this medium. The user must understand who or what they are in the context of the story; the more immersive the experience, the more impact the VR experience will have. When filming in 360º the creators can now think of the camera as the user. This creative transformation allows a director to tell a story from a first person point of view (POV) where the user becomes a character in the narrative.

Computer-generated VR

CG VR is an immersive experience created entirely from computer generated or animated content. CG VR can be pre-rendered and nonreactive, similar to 360° video, or rendered in Real-time using a game engine. Usually, pre-rendered CG VR is created to be consumed on a device that does not allow for positional tracking. real-time rendered CG VR, however, allows the user to move around in virtual space. Headsets like the Oculus Rift and HTC Vive have positional tracking, enabling the user to interact with their virtual environment through movement and controllers. This is considered true VR.

There are hybrids of VR that visually combine 360º degree video and CG to create an immersive experience using a blend of the two. Some of the more exciting VR content is being created using this approach.

Augmented reality

AR is an overlay of created content onto the real world. The CG sensory input can be sound, video graphics or geo-positioning (GPS) data. Today, real world content and CG content are not interactive, however research is quickly advancing that will enable AR to become interactive with reality.

As is sometimes the case, stories need to wait to be told until the right technology comes along to tell the story. Such was the case with Defrost. Furious M has been working with Feral Dog Production’s Tanna Frederick and her team on Defrost, a unique 12-part series directed by Hollywood legend Randal Kleiser.

Kleiser originally shot the first episode in 360º using a dual GoPro rig. Defrost is a liveaction 360º video told in the first person POV, so the viewer becomes the main character in the story: a 30-year-old woman who has been cryogenically frozen and is reanimated 30 years later. In this VR story, the protagonist is paralyzed and unable to talk due to atrophy. Viewers tend to look down and see their body in the wheelchair, their brains slowly connect and they virtually become this character, also known as experiencing ‘embodiment’.

While the 360º GoPro rig gave a sense of presence, Kleiser was after the feel of a stereo 3D – VR experience, and opted to shoot the remaining episodes using an early version of the new NOKIA OZO.

The production and post production of VR and 360 videos require meticulous planning and control. While the right camera choices are critical to the story, what the storyteller is looking to communicate in VR is reflected through the entire production’s execution, data management, pipeline, stitch workflow and file finishing.

Camera choices and rigs range from two camera GoPros to multiple RED EPICs and Light-field technology. It’s important to understand what you gain or lose in choosing your image capture system.

For example, RED EPICs will provide the filmmaker with better grain, image quality, control over exposure and temperature, and the ability to genlock cameras with time code insync versus a GoPros setup.

When shooting VR on set or location, production teams work with a cluster of cameras pointing in multiple directions or a single camera with a nodal head.

The visual capture acquired from these cameras needs to be reconstructed and stitched to a stereo pair, one for each eye in a HMD.

The new dimensions of VR capture expand the complexity of production planning and post, and as a result VR requires an updated workflow. While similar to existing 3D pipelines, VR requires additional considerations. At its core, the stitching process involves the removal of any image distortion introduced by the lens, matching common features between each camera’s captured footage, aligning all the frames into lat-long space and exporting the camera files for further compositing.

Many automated stitching applications are intended to ingest footage and then deliver a final, web-compressed video for distribution. However, very few offer a solution for source material with a higher bit-depth or dynamic range.

The methods for creating a manual stitch can vary immensely depending on software, artist skill, and choice of cameras. Given the complexity to synchronize, match, align, correct, stabilize and stitch footage, choosing an automated or manual stitching workflow will depend on how much control and flexibility is desired.

Presently, there are several stitching software choices available in the marketplace such as Videostitch, PTGUI, Autopano and the Foundry’s CARA. However, there remains a strong need in the industry for an automated stitching solution to reduce the labor intensive stitching process.

AMD recently announced ‘Project Loom’ a new real-time 360º video stitching technology and is collaborating with Radiant Images in Los Angeles. The Project Loom stitch solution will be able to pull files from up to 24 camera feeds at 1080p and 60 FPS, then stitch and output the resulting 360º video at 4K resolution and 30 FPS to VR headsets and mobile devices.

Radiant will offer “new solutions to clients who need live streaming of VR and multi-cam capturing from virtually any camera system, without the need of massive computing power” according to the company’s Michael Mansouri. The ultimate goal will be to create a much more dynamic cinematic experience for viewers.

Where are immersive experiences headed? And what is the end game? As creatives begin discussing VR projects, one of the great technical advances is AR, which is beginning to emerge in practical applications. The recent popularity of the mobile game Pokémon Go is an indication of the scope of the opportunity for AR. The combination of VR and AR or mixed reality format for storytelling has extraordinary potential.

—

Click here to translate this article
Click here to download the complete .PDF version of this article
Click here to download the entire Spring 2016 M&E Journal