Although you’ll never notice it, your eye doesn’t actually see everything in the high resolution that you’re used to. In fact, the human eye only sees a small portion of its field of view (FOV) in high res; detail in its peripheral vision is significantly reduced. In short, foveated rendering is a technique that uses eye-tracking technology* (read on for one an interesting exception) to mirror this effect in VR by only rendering the area that the user is looking at in high-res.
Here at ZeroLight, we create a range of real-time rendered experiences for automotive companies, and VR is a big part of what we offer. In order to deliver the highest possible quality in these projects, we are always looking to include the latest and greatest advances in our R&D work, so foveated rendering has of course presented an exciting opportunity for us! This Insights article explains how we adapted our approach to achieve foveated rendering in three different ways across a series of projects.
Foveated rendering v1: StarVR One, 2018
One of the most hotly anticipated VR head-mounted displays (HMDs), the StarVR One boasts an incredible 210⁰ horizontal FOV, a display with 16 million sub-pixels (roughly 5K resolution), and integrated Tobii eye-tracking tech!
Now, rendering a highly detailed scene on the HMD’s 1,830 x 2,928 display at 90fps is already a challenge, but to create one of ZeroLight’s true-to-life automotive experiences, we have to reduce pixel aliasing artefacts as much as possible. As a result, the actual render resolution is much larger: when using multi-sampled anti-aliasing, we render the scene at 4x the display resolution!
State of the Art VR Showroom Experience
Even though modern GPUs are very powerful, they are always going to struggle when rendering this resolution at 90fps with less than 20ms latency. The last thing we want to do is decrease the image quality, so we turned to foveated rendering in this project to help reduce the render cost by rendering everywhere the user wasn’t looking in lower resolution.
So, how did we do it? Well, we actually rendered the scene an additional two times: firstly, to the lower-resolution, wide-FOV target for each eye (where the user is not looking), and, secondly, to a focused, high-resolution render target for the fovea area (where the user is looking). The image below shows the two rendering configurations we support: 4 views for high-resolution wide-FOV and 4 views for foveated rendering.
Obviously, for this to work, we need to know exactly where the user is looking. This is where Tobii’s eye-tracking tech comes in. We used this hardware to track eye movements and calculate the optimal location of the high-detailed render area. The result? Take a look for yourself below! This VR capture shows the area that is rendered in higher detail – just look at the difference in clarity between the ‘S’ and the ‘V’ in the number plate!
But the impact is more than just visual: foveated rendering greatly reduces the pixel cost. Using Intel GPA, the two captures below show the performance difference when rendering the 4 viewports with equal pixel density split over the wide-FOV (~14ms) vs. with the foveated viewports (~11ms).
Updated for the HTC VIVE Pro Eye – 2019
So far, we’d relied on rendering multiple views of the scene per eye. While this does a great job of reducing the pixel fill rate cost, it adds an additional CPU overhead and added vertex cost due to the extra render pass.
As a result, we were keen to update our foveated rendering technique, and the new VIVE Pro Eye presented a perfect opportunity! As Announcement Partners for the HMD, we got early access before its launch at CES 2019, so we got straight to work on a totally new technique using NVIDIA’s latest Turing GPU, the Quadro RTX 6000. A new feature of the RTX GPUs meant we managed to avoid the previous overheads by using Variable Rate Shading (VRS), which enabled the application to render to a buffer at varying pixel densities.
Variable pixel density is the optimal rendering method for foveated rendering because the draw thread no longer needs to process additional scene data to render multiple passes for the foveal and peripheral views. This removes a large part of the CPU overhead that we experienced in the StarVR project.
A key part of this was knowing exactly where the user was looking. Once again, Tobii’s eye-tracking tech was the answer! New for the VIVE Pro Eye, this hardware helped us map the position of the user’s eyes, and we used this info to update the VRS mask each frame.
To push render quality to its maximum, we used a technique called supersampling where we rendered at a much higher resolution than the device’s native display. This was then filtered to produce a very detailed and smooth image without aliasing artefacts. In this project, we used 9x supersampling, meaning we rendered at 9x the resolution of the VIVE Pro Eye. Without using foveated rendering, this would have pushed the frame rate of the experience down to about 30FPS, but foveated rendering greatly reduces the number of pixels that need to be supersampled. As a result, we were able to hit a constant 90FPS while achieving significantly improved visual quality.
Hardware-based foveated rendering on the Varjo VR-1
As promised, there is one (partial) exception to foveated renderings reliance on eye-tracking, and it’s currently unique to the Varjo VR-1. Now, this headset hardly requires any introduction: anyone with even a passing interest in VR will have heard about the incredible power of the VR-1 after its launch earlier this year. And the industry reaction is with good reason: its human-eye resolution display is absolutely game-changing for enterprise solutions! Naturally, we had to get our hands on it.
So, the interesting thing about the VR-1 is that it actually uses a hardware-based foveated rendering effect to create human-eye resolution at the centre of the display when the user looks directly ahead. The effect is simply incredible – you can look at fine details and even read small text just as if you were in the real world! Take a look at the comparison of the resolution in ‘standard’ VR (on the left) with that in the VR-1 (on the right) below.
The integration of the VR-1 is actually just like any other HMD, but rather than rendering 2 views – one per eye – we render 4. This is similar to the foveated rendering technique we developed for the StarVR One, but without the eye tracking as the high-resolution fovea area is fixed in place by the hardware. It’s worth pointing out at this point that the VR-1 does have built-in eye-tracking technology, but we only used hardware-based foveated rendering in this concept.
How do we render this additional view without the CPU overheads we found with the StarVR One project? Well, for standard two-view VR headsets, we use NVIDIA’s Single Pass Stereo (SPS), which instances the view data from one eye to the other, removing the requirement to go over the scene hierarchy twice. Rendering the 4 passes of the scene that the VR-1 requires, however, would have a large overhead on the CPU.
We first got around this problem using 2 SPS passes, which halved the CPU rendering overhead, but to further optimise the performance for the VR-1, we integrated a feature of the new NVIDIA Turing GPUs: Multi View Rendering (MVR). This feature is similar to SPS, but it has support for up to 4 viewports in a single render pass. The typical use case for MVR is actually ensuring optimal rendering for a wide-FOV headset (like the StarVR One, for instance). Check out the image below for an example.
Image courtesy of NVIDIA
Foveated rendering is an incredibly useful and exciting tool, enabled by this new high-powered generation of VR headsets aimed squarely at the enterprise sector, the Varjo – VR1 & the HTC Vive Pro Eye. We’re very excited for what the future holds for enterprise VR.
