“Feelings and emotions are the very foundation of life” says the scientist Andreas Weber in his book The Biology of Wonder. His research makes a case for something that most storytellers know at heart, but scientists have been struggling to embrace – that humans (and other living beings) are creative agents fueled by meaning and expression, and that subjectivity and imagination are necessary for our existence.
To be human is to feel. And telling a story that does not evoke emotion is the same as not telling it at all. We know that, and yet, it is hard to bring this notion into the digital world. Just as science has a hard time to study and understand subjectivity, technology falls behind when offering ways for storytellers to take proper advantage of emotions.
If anything, this year has shown us that digital interactions disappoint when compared to real life. But the same digital interactions are enabling us to social distance for so long and to feel somewhat connected and engaged during these times. So there is a gap to be bridged, and state-of-the-art technologies are being developed to help us do that.
Volumetric video is one of them, and it has come to transform how we tell stories in immersive experiences. This is the only technology that brings real people — with their movements, facial expressions, voice and authenticity — into someone else’s reality, in 3D.
How does volumetric video work?
Volumetric video is the capture of a human performance in 3D. That means that it’s possible to capture a moving person in full 3D using a combination of regular cameras and infrared sensors. With this combination, we’ll not only have the regular RGB data but also the 3D data of each pixel in space. We’ll be able to reconstruct the subject in a tridimensional form. There are no motion capture suits or body-sensors involved, everything happens via cameras, sensors, and algorithms.
There are currently three main ways to capture a volumetric video:
- High-end Studios: these studios are composed of the best possible cameras and sensors available, and they can create a detailed and high-quality capture. Examples of these systems are 4D Views, TetaVi, and Microsoft Mixed Reality Studios.
- Do-it-yourself: it’s also possible to capture a volumetric video by yourself using off-the-shelf sensors such as the Kinect Azure, and a capture software such as Depthkit or Brekel. This form of capturing is much more accessible, only demanding that you’re not afraid of learning how to do it yourself.
- Cloud-processed Volumetric Video: in this case, you can use a rig of only regular RGB cameras to capture the subject from multiple angles, without the need for special sensors. Then, you upload the 2D videos to a cloud platform that will transform them into volumetric videos using machine learning techniques, and send the hologram back to you. This is the newest possibility on the market, provided by companies such as Volograms and MOD Tech Labs.
Why volumetric video is the next frontier for storytelling?
The fact that we can now carry all the realness of a living being into the digital world in 3D, means that we can convey emotions with it as well. Nonverbal communication transmits feelings and attitudes, and it is of great importance to fully interpret the message we receive. They include facial displays, corporal performances, and also vocalizations; all of which are recognized and used in communication across cultures. Research indicates that emotional expressions shape behavior and influence judgments and decision-making in a range of domains. Making it necessary for any form of impactful storytelling.
Simply put: emotions are the gateway to engagement and digital storytellers should rely on technologies that authentically portray the expressions that only real humans can do.
As shown above, volumetric video is the right technology to do it. And as in any other XR experience, the viewers count not only on the emotional engagement enabled by this technology but also with physical interactions in the experience. Get closer, circle it, tap the screen — in essence, you can choose your journey and see how the volumetrically captured person will react to it.
Volumetric capture in XR is a two-way street. The story is not being told to a passive listener. It is being actively created with the viewer.
Volumetric video is right here, for anyone to use it
Volumetric video is a very new and innovative technology, and up until now, only specialized software engineers were able to create stories with it. Extremely complex game engines paired with heavy coding were the only option for those who wanted to try it out. Making it impossible to be adopted by most professional storytellers out there, such as advertisers, video-makers, journalists, content creators, artists and alike.
That’s why we, at Sense of Space, are determined to make volumetric video more accessible to all creatives. Sense XR Studio, our first product, allows incredibly easy importing and editing of volumetric assets together with audio. And it also offers users the possibility to create interactions in an intuitive way. Culminating in an experience that can be easily exported to WebAR and other platforms.
Although the product is still under development, we have opened a part of it as a free alpha release. So anyone can test it out and offer their feedback to us. As a company founded by creatives, not technologists, it is of great importance for us to be closer to our fellow digital storytellers, and learn about the challenges and necessities we face in this field.
Engaging digital storytelling depends on our ability to make use of meaning, feeling and expression. We are working hard to make XR experiences more authentic, to bridge the gap between what is real and what feels real. The gap that once closed will allow our stories to run free around the world.
Volumetric video is here to stay. Create with it.
