← Portfolio

Project · January 2020

Live Action VR

A branching VR narrative that composites a live-filmed actor into a synthetic 3D environment instead of scanning and rigging a digital double.

Role
Unity engineer, real-time compositing pipeline
Tags
Virtual Reality, Unity, Shader, Real-time Compositing
3D modeling software viewport showing an untextured gray spacecraft model with a central cockpit pod, long twin wings extending outward with cylindrical engine pods, a tall vertical thruster at the rear, and a landing leg extended below. Viewport overlay displays mesh statistics and camera settings.

Cover photo courtesy of Phat Nguyen.

Live Action VR was an ICAT Mini SEAD-funded project exploring a different path to virtual characters. Instead of 3D-scanning an actor, rigging, then blending textures and animation in real time, the project captured a live performer on video and composited them directly into a synthetic 3D environment. The goal was an interactive, branching narrative where a live participant, wearing a VR headset, could converse and navigate scenes with a real actor rather than a scripted digital double, raising the emotional stakes of the experience while keeping the setting entirely virtual. Awarded Board Choice at VT ICAT Creativity + Innovation Day 2020.

The project drew on an interdisciplinary team spanning Computer Science, Cinema, and Visual Arts. On the production side, a dual-camera 3D rig filmed the performer in HD against a green screen. On the technical side, that video feed had to be streamed in real time into a Unity-powered game engine and composited into the scene, with a shader removing the green screen so the performer read as part of the environment rather than a video overlay on top of it — a genuinely hard real-time compositing and latency problem, compounded by production details like lighting the actor to match the virtual scene’s lighting.

My contribution was the Unity-side implementation of that pipeline. I worked from the physical camera rig setup to integrate two Blackmagic Micro Studio cameras as the live stereoscopic capture source, then built the Unity scene spanning the story’s several distinct environments. The actor was chroma-keyed with my shader and composited into the environment live. I mapped the left and right camera feeds to the left and right eye of the VR headset respectively, preserving correct stereoscopic depth and pupillary distance so the composited performer appeared properly positioned in 3D space rather than as a flat video plane in front of the viewer.