There’s been a lot of talk and hype in the XR industry surrounding devices that enable users to interact with the virtual world. Rumours about Apple glasses are vigorously discussed on social media and the success of the Oculus Quest 2 is a common topic in mainstream media. Also, the content side of the house gets its fair share of attention not only for the everlasting Pokemon Go but also all kinds of consumer AR apps like Snap Lenses or enterprise cases such as VR Training. While of course devices and content (and the respective content enablers and frameworks) play a crucial role for truly immersive experiences, another key success factor for XR is lagging behind in terms of attention: the network and cloud infrastructure, i.e.
- The low-latency, broadband networks to transport the massive amounts of data required for high-quality real-time experiences
- The (edge) cloud infrastructure provides the computing power to render graphics with datacenter GPUs and run analytics workloads to have a real-time user and scene understanding
5G will be king!?
Surely you’ve come across the claim of 5G solving almost every technical problem including those of today’s XR experiences. And while this is obviously not the case in this ubiquitous manner there is some truth in the fact that broadband, low-latency networks play an important role to power truly immersive XR experiences. A high-quality volumetric stream may easily consume 50 or even 100Mbps (Sources: Intel, Qualcomm) and high-res 3D models and textures can take a couple of 100 Mbytes. Preloading the data on the device can be done (if the device has sufficient storage) but even then the data needs to be transferred over a network and the user may not want to wait for a long time to finish. Therefore, low-latency, broadband networks like 5G or WiFi6 are required for high-quality XR. 5G is especially important for truly mobile cases like many AR use cases while stationary use cases (as many VR cases) will likely more utilize WiF – in both scenarios it will be through a mix of connectivity options.
Getting the devices connected
With the advent of WiFi6 the Quality of Service required for XR experiences is becoming feasible and we will soon see this replacing USB cable to tether XR devices like the Nreal light. Oculus announced already the availability of Oculus Air Link as an experimental feature. Embedding different connectivity options into (head-worn) devices is not an easy thing due to technical and cost challenges. According to a recent whitepaper by Huawei and Strategy Analytics out of “63 AR device models on the market (primarily enterprise devices), 48 have embedded Wi-Fi connectivity, one has 4G LTE and one has 5G.” With smartphone tethered AR (and also VR)-devices, this number will soon go up and build a transition towards head-worn XR devices with 5G connectivity built-in – which is yet another challenge for the engineers to optimize form factor, weight and height dissipation.
“The speed of light sucks” (John Carmack)
The speed of light in fibre is around ⅔ of the speed of light in a vacuum. That translates to around 200km per millisecond. If you think that’s pretty fast (and as you know it’s as fast as it can be) consider that a few milliseconds of a too high latency can screw up a great XR experience. The motion-to-photon latency, that is the time from an event like clicking a button until a photon reflecting that action (e.g. a pixel change) is hitting the user’s eye, shouldn’t exceed 5-20ms. While there are techniques like prediction or speculative execution to compensate for some latency still the allowed latency budget for highly interactive content remains pretty low. This is the reason why we do not only need broadband networks, but also low-latency networks for a highly immersive XR experience. As the motion-to-photon latency is determined by every element along the path including the computation on the device and the cloud it is important that all those elements in the chain are highly optimized to reduce latency.
The latency in a is driven by four factors: protocol latency, errors causing retransmission, congestion and distance. All of them are tackled in different ways but the distance (remember the 200km per millisecond) problem is tackled by what is called today ‘edge computing’:
The edge is a continuum
The concept of cloud computing is well-known to everyone for being the virtually unlimited compute and storage driving most of today’s modern applications. As XR devices are even more constrained in terms of weight, power supply and heat dissipation than mobile phones and enabling and user-facing XR services are computationally very demanding this concept is also crucial for XR.
The Metaverse is made out of a lot of data and that data needs to be stored and processed somewhere. Constant streams of sensor data from XR devices need to be analyzed in real-time using machine learning and other computational demanding algorithms, e.g. for cloud-based mapping and localization (as done by 1000realities or immersal). To achieve high-quality rendering with raytracing you need to render in the cloud (see e.g. Holo-Light, Mawari, QuarkXR or the NVIDIA CloudXR SDK).
The well-known public cloud infrastructures do enable all of that, but in many cases, they are simply physically too far away from the devices to keep the motion-to-photon latency within the given budget. The solution is edge computing, which means distributing the cloud infrastructure across the geography and often placing it directly into the operator networks. That enables the low-latency to compute offload cases and also a scalable, high-frequency synchronization between devices in the same region, another key enabler for multi-user shared XR experiences. This makes it possible to not only share your video while playing a smartphone game live with your opponents but also to augment it with visual effects and run further analytics on it – all in real-time as demonstrated by our partner Proxy42.
Putting the puzzle pieces together
Truly immersive XR experiences require the highest quality along every element of the chain: high-quality content and enablers, high-quality devices but equally high-quality networks and edge cloud infrastructure.
If one of these elements is missing it will break the whole experience. Moreover, these elements need to be integrated perfectly and need to be optimized hence, a close collaboration along the stacks and value chains is highly important.
TangibleXR can help you to identify and integrate the best of breed solutions along the chain to build truly immersive experiences to unlock the full potential of XR. If you have one of these puzzle pieces we also help you to position yourself properly into the ecosystem and open up new business opportunities. Let’s talk!
For more industry insights visit this link.