Insights: Case Studies

Bringing the Star Wars Universe into VR: The Making of Secrets of the Empire

Alternative Text

Ben Snow | Visual Effects Supervisor & Key Creative | ILMxLAB

25 Jan 2019 | 10 min read

About Secrets of the Empire and its Creators:

Star Wars: Secrets of the Empire is a hyper reality VR experience designed to give the participants, our guests, the opportunity to step into the Star Wars universe.  With a group of up to four friends, they physically navigate through a Star Wars landscape and corridors of an Imperial base, interacting with objects, characters and each other, to accomplish the mission given to them by Captain Cassian Andor.  The virtual environment is mapped one-to-one over physical walls and practical props that can be touched and interacted with, like buttons and blasters. The visuals are real-time graphics seen through virtual reality headsets.  Through The VOID’s proprietary technology, guests feel the heat and wind of a lava planet and the vibration of their spaceship, as they walk through the corridors of the base and even smell ash in the air.

Star Wars: Secrets of the Empire was designed as a collaborative project between ILMxLAB and The VOID, that showcased ILMxLAB’s best-in-class immersive storytelling and cinematic quality, real-time computer graphics, and The VOID’s state-of-the art “hyper reality” technology, physical stages, multi-sensory effects and experience design expertise. At ILMxLAB, our mission is to have guests literally ‘Step Inside Our Stories,’ and this unique prospect with The VOID was a perfect opportunity to further this goal. For this project, our overall objective was to use hyper reality to create an original, adventurous Star Wars story set in the classic era, providing guests an experience that is both crafted for the medium from the ground up and provides the ultimate Star Wars wish fulfillment.

Upon release, Star Wars: Secrets of the Empire was met with critical recognition from both the press and the public, winning the category VR Experience of the Year at the second annual VR Awards in October.

Creating a Compelling Narrative:  

Guests begin the experience by viewing their mission briefing from Captain Cassian Andor from Rogue One: A Star Wars Story via 2D video transmission: under orders from the fledgling rebellion, uncover the contents of a mysterious crate that the Empire is being transported to the planet Mustafar.  Then, the teams of four don the VR headsets and haptic vests that they’ll wear throughout the hyper reality experience. When guests lower their VR visors, the Star Wars universe appears all around them and they find themselves in the cargo hold of a spaceship digitally disguised in their selected stormtrooper gear.  They are beckoned by the droid K-2SO into the cockpit, where they take in an ominous view of the lava-drenched planet Mustafar on their approach to the facility. As it lands, they walk out onto a flying skiff platform and enter a secret Imperial facility.  When they move through the enemy compound, guests encounter Star Wars sights and characters, and pick up blasters that they use to defend themselves after their presence is discovered by Imperial forces.

Guests in Void Gear

Guests in VOID gear

Bringing an Evocative Virtual World to Life in the Early Stages of Development:

Within the broader constructs of our storytelling initiatives at ILMxLAB, our team saw this as an opportunity to set up the in-home experience we were creating with Vader Immortal: A Star Wars VR Series and chose to set the adventure on the planet Mustafar. ILMxLAB, the Lucasfilm Story Group and The VOID collaborated to create a list of exciting story beats that would be compelling on The VOID’s platform.

In order to better understand the right experience flow, The VOID actually had us place sticky notes with these key story moments written out on the floor, which was taped out to approximate the eventual stage this experience is now mapped to, here in our offices at Lucasfilm.  After many revisions, we did a spoken walk-through over the taped-out floor that convinced us that we had a compelling story that people would want to experience. We built a greybox model of the project in Unreal Engine 4 which we used as the basis for concept art. This enabled us to expand upon the roughed out physical space to show what guests would see in virtual reality.  During this time, we also refined the story outline and wrote a script with Tracy Hickman and David S. Goyer. The greybox, concept art and script, provided a solid base for moving into production, and demonstrated to our internal stakeholders and teams that we were creating an exciting, new, canonical Star Wars immersive story.

concept art

Star Wars: Secrets of the Empire concept art by ILMxLAB

 

concept art 2

Star Wars: Secrets of the Empire concept art by ILMxLAB

 

Challenges we Faced Balancing the Narrative with Compelling Objectives:

Communicating our story while providing a clear and compelling objective for our guests, was one of the biggest challenges we faced.  The VOID’s experience in designing for hyper reality was critical in anticipating potential problems.

For example, we knew that many people would be experiencing hyper reality for the first time and would be distracted by the sheer novelty of the medium. Also, since guests communicate and travel in teams; we knew that they would want to interact with each other as well as with the experience. Our team had to anticipate this, and provide time within the experience for this to naturally occur. In addition, we had to find creative ways to ensure that the guests listened and paid attention to the story or action at pivotal junctures.  

We ended up eliminating lines of dialogue simply to enable teams to pace their own experience.  The objective for guests is to track a mysterious cargo crate and find out what is inside. The VOID refers to this concept as the “critical thought”. The objective is outlined in a pre-show video, emphasized in holographic transmissions after the guests entered VR, and then the crate is shown as they follow it through the facility they are infiltrating. We allow time so that the guests can chat and comment on what they were experiencing, and still not miss anything critical. For example, after K-2SO invites them into the cockpit of the ship we space out his and another character’s dialog to allow guests time to comment to one another.  We direct their gaze to certain areas of the scene by audio cues, lighting and room design. In one particular moment, we even seat the team so they are looking exactly where we need them to and can hear what the characters are telling them.

Creating an Authentic Experience and Matching the Look and Feel of the Star Wars Universe:

For the visuals, performance and animation our challenge was to match the feel of the Star Wars universe as seen in the films as closely as possible.  For example, to develop our main characters we started with ILM’s film assets and optimized them for virtual reality. We developed real-time materials in Unreal Engine 4 that matched the movie versions.  We also used as much of the same rigging and secondary movements like the pistons on K-2SO as we could manage in the real-time medium. Additionally, we used concept artists who had worked on the films, and worked closely with the Lucasfilm Story Group to ensure our story fit firmly within Star Wars canon. David S. Goyer, one of the writers on the experience, provided important contributions for the script. In keeping with our goal of creating an authentic experience that seamlessly matched the look and feel of the Star Wars films, a crucial element in the development of the project was Alan Tudyk (K-2SO) and Diego Luna (Captain Cassian Andor) reprising their roles from Rogue One: A Star Wars Story.  

Alan Tudyk reprised his role as K-2SO from Rogue One: A Star Wars Story

 

Testing and Refining the Project in the Later Stages of Development:

The dispersed companies and facilities were other challenges throughout the development process.  As a part of Lucasfilm, ILMxLAB is based in San Francisco and The VOID headquarters are in Lindon, Utah. As a result, we often sent team members to Utah from San Francisco for regular build tests and creative design meetings.  Meanwhile, The VOID team experimented to develop the physical elements and interactions. They tried numerous scents to create various smells in the experience. A burning wire smell was tested and subsequently rejected because it was too unpleasant.  After several trials, we settled on the smell of wood smoke to evoke the ash-laden environment of the lava planet. Our audio team was faced with both the directional audio needs of VR and with driving the vest haptics and floor transducers used to provide rumbles and vibration throughout the experience.    

As soon as we had a solid greybox populated with blocking animation and interactions; testing started at The VOID. First, with friends, and then with members of the public.  We observed them in the experience and had them fill out surveys afterwards, in order to identify and fix story comprehension issues, potential discomfort, or lapses in action.

The project was created in less than a year, starting in early 2017 and opening to the public in December 2017. The first few months of development were focused on story and pre-production with a minimal crew to create the first greybox: the initial crew included the Lucasfilm Story Group, the creative team at The VOID, and a lead designer and an engineer from ILMxLAB.  Towards the end of spring, we integrated our entire team, with a goal of hitting Alpha by mid-summer, meaning that the experience would be functional, and include all animation, environments and most gameplay. We planned Beta by the fall, which would entail polish, optimization and debugging. We had a core team of about thirty at ILMxLAB with twenty others at The VOID, along with ad hoc support from other ILM artists.  Epic also contributed to the project, lending Unreal Engine experts as needed to ensure that the experience was optimized in their engine. We used an agile/sprint project management approach and tailored our work to the skills and size of the development team.

Releasing Secrets of the Empire:

Star Wars: Secrets of the Empire opened in three locations worldwide, with two locations adjacent to Walt Disney Theme Parks in both Anaheim and Orlando. Reactions from consumers and press were positive and enthusiastic; and The VOID has continued its rapid expansion to include nine additional experience centers globally with more forthcoming.

The project was a success and gratifying for the team. Key success factors were:

  • Developing a close and honest collaborative process at every level between ILMxLAB, The VOID and Lucasfilm Story Group.  
  • Emphasizing story, having the guests feel it was their story, and particularly their Star Wars story, while staying true to the canon.
  • The ability to work with film talent for role reprisals (Alan Tudyk and Diego Luna), and award-winning writing/production talents (Tracy Hickman and David S. Goyer).
  • The use of Unreal Engine 4 – both for the quality of the real time graphics and for the support Epic gave us in developing the project.
  • Providing Star Wars wish fulfillment for our guests, and developing things that were truly exciting to do in VR that played to The VOID’s strengths as a pioneering immersive virtual experience platform.
  • Starting our project with a strong outline, sketched out on paper and taped out on the floor, followed by greybox and script development before production started to drive concept art, blocking animation and engineering tasks.
  • Careful production management utilizing Agile/Sprint with Jira for task management.
  • A small, talented collaborative core team drawing on outside expertise when necessary, and adhering to strict deadlines and schedules.

We believe that Star Wars: Secrets of the Empire immerses its participants in a Star Wars story in ways that are simply not possible in any other medium.  Like any virtual reality project it was a challenge, but one well worth taking, and one that we will continue to build upon for future projects.

About the author:

Ben Snow has worked at Industrial Light and Magic for the past 25 years, currently serving as Visual Effects Supervisor for both traditional film projects with ILM and immersive experiences with ILMxLAB. For ILMxLAB, Ben has worked as a Visual Effects Supervisor and Key Creative for Star Wars: The Force Awakens The Immersive 360 Experience and Star Wars: Secrets of the Empire. Throughout his career in film, projects Ben has been directly involved in have garnered 4 Academy Award Nominations, 2 BAFTA nominations and 8 Visual Effects Society Nominations.