Insights EDU

Utilizing the Inherent Impact of Mixed Reality

Alternative Text

Tobias Sievers | Creative Director | Luxoom

20 Jan 2023 | 8 min read

Abstract: How can common anchors in real and virtual components of XR showcases strengthen their impact and enable a consistent experience and storytelling? And how can experiential design parameters help deepen the impact of those showcases?

The Mediatecture Approach

Working day-by-day in experiential design, we think and design in terms of customer journeys. We challenge ourselves by asking how experiences can best be created in the spatial and temporal environment of target groups. We try to define spatial (physical) and temporal (event) anchor points to link our storytelling and visual work. Experiences that we create consider environment and context. They utilize those to create impact. Even though this is not rocket science, it’s worthwhile to ask, what is the mechanism behind it?

Under the title mediatecture a colleague and I published a textbook in 2018 that aims to provide an answer to this question. Mediatecture is an approach towards a medium that actively and virtually extends the experience space of target groups. Yet it does not rely on any virtual digital technology like VR headsets or AR apps. It rather measures the impact of spatial and temporal interventions in parameters of immediacy, relevance, elegance and efficiency.

In short, we can understand immediacy as a facilitator for the immersive potential of an experience. Immediacy asks for something to be felt rather than needing contemplation. Relevance talks about how well something connects to the intellectual state of the target group. Does it get their attention? Does it motivate them to follow the idea? Elegance describes how apparent and simple a creative solution is. In general: the simpler and more apparent, the better. Efficiency, on the other hand, acts as kind of a counterpole. It asks how much effort needs to be put in and what parts of a solution might be irrelevant for the impact of the result.

The Introduction of Virtual Experiences

Then came the day we started to play with virtual experiences. We had already implemented VR design in our design pipeline for experiential architecture. So, when we got asked to open our studio for a public design week, we took some of the projects and linked them together into a virtual showcase. We invited attendees to simply grab a VR headset and enjoy a tour through our work. To link the different projects in the showcase we decided we needed a lobby. On the one hand we wanted to give visitors with little VR experience a safe space to test navigation. On the other hand, we favored building an architectural access to the different projects (instead of simply jumping from one showcase to the next).

So, we virtually replicated our studio – the Luxoom Lab – which is a decommissioned funeral hall, and turned it into a lobby for the show. In the physically-real hall itself, we created a huge projection of a portal – the exact portal through which visitors could step into our different virtual projects when navigating in VR.

The Luxoom Lab with the projected portal as a starting point for the VR Journey. This hall existed in both the real and virtual worlds.

One of the showcases that was part of the journey – a real-time visual environment designed for the National Museum at Astana, KAZ for Exponentialgroup.

Besides the showcase being a success and a crowd puller, the attending staff started to witness a specific and peculiar behavior with the visitors. People didn’t simply take off the VR headsets when finished. They first navigated back through the portal into our funeral hall-cum-lobby to end their journey. Real and virtual spaces seemed to have overlapped in a way, participants in their own perception left the real space. They accordingly felt required to return to it before being able to leave. An effect that seemed to chime with what we described by the term of immediacy before.

The Question of Perception

When working on mediatecture we followed a constructivist approach to give an academic foundation to the kind of effects witnessed here. Small recap: Constructivists state more or less that all things humans perceive are solely synthesized in our human brains. There is no direct link for example from eye to perception, all is only accessible to us as a representation on a stage that our brains entertain. And this stage might use as much pre-conception as actual sensory input to construct what we perceive. As a side note: this all happens to make our perception stable and efficient. 

Normally our brain uses cross-references between different senses to determine what’s real and what’s not. In physical space we already have a very clear preconception that a concrete wall is solid and you would bump your head against it if you try crossing it in a way that is possible in VR. To a certain degree we follow these preconceptions also in virtual environments, where they might mislead us – but that is a different story.

Returning to our experience in our lab, we had the physical space made out of brick walls that every participant would assume to be solid. And we had the matching virtual representation of it – the lobby and portal within our virtual showcase. From what we know about how reality is constructed (the constructivist approach) and what we witnessed from the participants‘ reactions, we can argue the difference between what is really out there and what is augmented is not that easily discernible for our brains – especially when there are (preconceived) matches between real and virtual environments. Yes, once a participant puts conscious brain power to that question, those are easily discernible, but in the realistic circumstance that the conscious focus of the brain is occupied with the experience, the real and virtual blend quite amazingly.

Virtual Equals Real?

Let’s use this point to differentiate between augmented and mixed reality – not for the sake of a definition, but to use the two modes to pinpoint a subtle but important difference. Both modes combine the real with the virtual. AR is mostly used to produce additional information as a kind of layer on the canvas of reality. What we did in our lab is slightly different: We created a transition between reality and virtual while keeping elements of the environment stable – namely the lab hall. By keeping the environment stable, we can argue, we created a stronger bond between what was real and what was virtual.

What interests us here, when looking at mixed reality, is first of all the transition situation. At this point we made a first assumption: we can create a continuous experience and therefore a storyline between real and virtual. This works as long as we put our design-focus on those transitions. Speaking with the parameters we described: immediacy in the sense of virtual equals real is the key factor here.

Testing That Idea in a Product

Time to convert this idea into a product we could use to further test our findings. We pitched the idea with a client who manufactures complex machinery products. These are the kind of machines that keep their technical USPs well hidden inside their technical structures. The aim was to give the target group the feeling of being able to see those USPs, To see what happens inside the machines when in action, while keeping the threshold between real and virtual as minimal as possible. For this solution, instead of having to wear VR headsets, we relied on an iPad as a camera and tracking device connected to a rendering computer rig. Namely to address larger crowds while keeping the visual quality gap between real and virtual as minimal as possible.

And then it’s showtime – the physical machine is situated on stage while a large video screen shows the same machine virtually in action. In the first step, the virtual camera flies like a drone to the camera position of the iPad and smoothly transitions the image to the real machine. It matches the virtual machine in size, perspective and even visual lens distortion. The storyline then again and again creates transitions between augmented and fully virtual contents – testing our idea that repeated carefully designed mixed reality transitions can deepen the impact of the experience, instead of minimizing it.  The machine acts as the anchor between the two worlds.

Using the iPad to take a look into a complex technical machine, and reveal USPs in action.

The Results

Seeing the audience‘s reactions, our assumptions held true. Talking to the audience, they did not discern between the real machine and its virtual twin. The topics discussed and benefits recounted were not abstract but connected to the real machine on stage. To put it into simple terms: in the perception of the audience, the mixed reality contents had become part of the machine. More detailed information about our solution can be found on the product website.

Mixed reality done right, inherently holds great impact for communication and storytelling. The four parameters that we took from experiential design and mediatecture can help to optimize XR solutions. Immediacy inherent in mixed reality as we understand it gives us the basis to design impactful transitions that weave a tight fabric of the virtual and real. Relevance is created when the virtual reveals how technology leads to benefits,  as well as when story and visuals of both virtual and real build a joint, conclusive picture. Elegance is in the method. For example, the simplicity of the iPad as a daily tool keeps the technical threshold accessing the solution visibly low. And the simplicity of the technological approach also adds to efficiency. It keeps the focus away from otherwise-required specialized hardware and ultimately on the machine and story.

About the author:

Tobias is creative director, lecturer and author in the field of experiential design, spatial media and XR with 20+ years of experience. He taught digital design at the China Academy of Arts and exhibition as well as media design at the University of Arts, Berlin. With "Mediatecture - a handbook“ he co-authored a key study book on creative spatial media design, bringing strategic design criteria to the field. As lead creative of Luxoom studios, he is the driving force behind a broad portfolio of global experiential design and production projects for cultural and commercial brands. Central to his work is the aim of ever further bridging the gap between online and offline experiences by understanding human perception and interaction.