Contents
Introduction
When and how to use it?
About this sample code
Using the sample
Conclusion
Use of the code snippets present within this page is subject to the terms of the Mali OpenGL ES SDK for Android EULA.
Introduction
Physically, your eye is made in such a way that only 5° of it can see the full detail of any given view. The further away you get from the point where you are focusing your vision, the less detailed it will appear. Foveated rendering, as discussed here, refers to using varying rendering quality to improve rendering time by only producing the highest resolution where it matters for the viewer.
VR is the big current use case for foveated rendering, so the following discussion will be in the context of VR. We won’t make any assumptions of gaze tracking; however, this rendering solution can easily be adapted and improved using such a device.
For more information on Multiview, please refer to the following blog: Optimizing Virtual Reality: Understanding Multiview
When and how to use it?
If the application is fragment bound but not vertex bound and the platform supports 4 views for the Multiview extension, it is possible to implement a foveated rendering algorithm as per the one in the sample. Foveated rendering allows you to reduce the detail in the area that is outside the “main” area of focus of the viewer’s eye. Consequently, this allows you to use a lower resolution framebuffer to achieve very similar quality with lower fragment shading cost.
The idea behind the technique is to render 4 views instead of 2 but using lower resolution framebuffers compared to the 2 views case. 2 of the views will still render the scene as usual while the other 2 will render the scene with a reduced field of view. Reducing the field of view causes a zoom effect on the central part of it and causes the rendered image to have more detail in that area. This makes it possible to have the same resolution for the central part of the view that we would have if we were rendering on the higher-resolution framebuffer used in the classic multiview sample.
The resolution of the framebuffer used, the field of view used for the zoom-in view, and the quality of the final image are directly correlated. The more you lower the resolution of the framebuffer, the smaller the high-res area will be and the lower the quality in the surrounding area.
This table shows the number of pixels saved by foveated rendering:
Technique
Texture resolution
Pixels rendered
Savings in pixels compared to 2-View Multiview
2-Views Multiview
1024×1024
2097152
0
4-Views Foveated Rendering
30% reduction
716×716
2050624
46,528 pixels saved
50% reduction
512×512
1048576
1,048,576 pixels saved
To achieve the benefits of foveated rendering the application needs to:
Be fragment bound: In our sample code we simulate that using a very complex fragment shader for the plane
Not be Vertex Bound: In our sample code the vertex load is minimal
This last point is important as using 4 views will increment the amount of work done in the Vertex shaders as explained in the previous section. The Vertex Shader that is using Multiview will have a for-loop for 4-view and if it does a lot of computations using the view index as data selector, that computation will be done for each of the views.
Quality is also an important factor. Reducing the resolution of the framebuffer too much will make the resolution of the surrounding area too low compared to the high-res area, this makes the difference discernible to the user and is therefore unacceptable.
About this sample code
This sample code creates the scenario needed to make foveated rendering effective as an optimization for VR.
At initialization, the sample code creates a framebuffer based on the ratio specified, for example a ratio of 30% will create a framebuffer of 360×360. After that, the final framebuffer is created with the resolution of 2048×1024 (1k per eye). This framebuffer will be the render target when combining the separate 4-views into a single stereo image.
The sample also renders a mask that is used to exclude part of the surrounding area from rendering since it will be overridden by the high-res area. This is done by rendering only to the depth texture, thus making the following pixels fail the depth test.
This is another optimization that further increases the savings you will get when applied.
Technique
Texture resolution
Pixels rendered
Savings in pixels compared to 2-View Multiview
2-Views Multiview
1024×1024
2097152
0
4-Views Foveated Rendering
30% reduction
716×716
2050624
46,528 pixels saved
50% reduction
512×512
1048576
1,048,576 pixels saved
4-Views Foveated Rendering
Stenciled
30% reduction
716×716
1,439,214
657,937 pixels saved
50% reduction
512×512
833,119
1,264,032 pixels saved
After being rendered, the 4 views go through a combination phase where the 2 surrounding areas are up-sampled to fit on the final framebuffer and the 2 high resolution areas are copied on a 1:1 basis.
To summarize, for each frame the sample will execute the following operation:
Clear all framebuffers
Use the mask specified above to initialize the depth buffer – This is achieved by drawing a quad positioned at the near-plane distance from the camera, for the high-resolution views this will look like a full-screen quad while for the surrounding view it will look like a small quad covering the area that will be covered up by the high-resolution view
Draw the geometry with heavy fragment shaders
Change the render target to the final side-by-side framebuffer
Set the viewport to cover the left eye area
Render a full screen quad that samples from the High-resolution and Surroundings views for the left eye based on the distance of the pixel from the centre of the image
Set the viewport to cover the right eye area
Render a full screen quad that samples from the High-resolution and Surroundings views for the right eye based on the distance of the pixel from the centre of the image
Apply any other post-processing such as barrel distortion
Using the sample
This tutorial requires additional resources to work as expected.
These resources are available inside the ”foveated“ folder and must be pushed on the device prior to the first execution.
Advised procedure to push these is:
This sample is made in such a way that you can test and experiment with different configurations. Available settings are:
REGULAR
This enables the regular rendering pipeline
MULTIVIEW
This enables the multiview rendering pipeline
FOVEATED
This enables the foveated pipeline
STENCIL
This enables the stencilling of the view in foveated configuration
RED
This will highlight the fovea in red
RATIO=X
This allows you to choose which reduction ratio you want for the foveated configuration
X being the inset reduction wanted, 0.1 is 100% or full screen inset and 0.5 is 50% or half screen inset
These needs to be defined in the file before the includes in the following way:
Frame per seconds will be printed in the console; you can easily get them using a logcat filter:
adb logcat -s “Foveated_Sample”
Conclusion
As we have seen in this sample, foveated rendering will help you mitigate fragment bound applications by trading number of fragments shaded for vertex processing cost. A careful analysis of your bottleneck is therefore necessary in order to ensure optimal benefits.
We didn’t discuss the benefits of eye tracking in this tutorial, but using such a technology can greatly impact the final outcomes as you would be able to reduce the size of the inset region even further.
I would also strongly suggest you considering using multisampling with the EXT_multisampled_render_to_texture version of the multiview extension. Multisampling will help you greatly reduce the amount of flickering on the low-resolution area as this flickering usually comes from aliasing issues.
The optimal configuration we would recommend for foveated rendering without eye tracking is 50% inset ratio with stenciling. That way the inset region is harder to clearly distinguish and provides a better end user experience.
The post Advanced usage of Multiview: Foveated Rendering appeared first on Mali Developer Center.