WebXR not supported on WebGPURenderer

[hybridherbst]

Description

As it seems that work on WebGLRenderer will go into maintenance mode to make more space for WebGPURenderer, it would be great to see an initial implementation of WebXR so we can start testing WebGPU as well.

We'd like to test and help, but I'm not sure how WebXR fits into the current architecture of the new system with two backends. I'd already be happy if WebGPURenderer ({forceWebGL: true}) had WebXR support (so, not actually WebGPU).

Reproduction steps

1. Use WebGPURenderer
2. Try to use XR
3. Note that APIs don't exist anymore (getSession(), getCamera(), isPresenting, ...)

Version

r167

Device

Headset, Phones

OS

Windows, Android

[Mugen87]

As it seems that work on WebGLRenderer will essentially stop to make more space for WebGPURenderer,

Sorry, I need to correct this statement since it isn't right. We do not stop the work at WebGLRenderer but just limit the scope. So there will be bug fixes and smaller new features but no groundbreaking refactoring or enhancements.

[hybridherbst]

No problem, adjusted the wording to "will go into maintenance mode", which is what I meant.

[hybridherbst]

From my understanding, there is no specification yet for how to use WebGPU with WebXR (@toji, if you want to chime in that would be cool!).

But what could likely be done is to add WebXR support for new WebGPURenderer({forceWebGL: true}), which would unlock testing and verification of the new backend structure for applications.

@mrdoob do you have an opinion on how WebXR should be changed/added to the backend(s)?
I'd be interested in exploring that, and I believe so would @rcabanier.
I think it would be great to get WebXR added to the WebGL backend for now, and then later once the specification is ready the WebGPU backend can get support as well.

[CodyJasonBennett]

This needs https://github.com/immersive-web/WebXR-WebGPU-Binding as there is no way to bind to the compositor from WebGPU, nor is it a good idea to mix WebGL and WebGPU on the same page, performance ramifications of a full copy aside. Also not the easiest with coordinate conventions differing, although not too bad with only two views or projections to correct -- quick solution here https://twitter.com/Cody_J_Bennett/status/1658786889577496579.

It would be nice if this wasn't so internal to the WebGL backend (like it is now with WebGLRenderer) as that makes it hard to maintain for texture code-paths specifically, and locks it down to future improvements like multi-view without a rather involved refactor, partially due to limitations or issues on Quest which may resolve in time (or be dropped for alternatives like old-school tricks using instancing/multi-draw or storage memory if you don't have a high number of indices -- remember indexed drawing is an important optimization for TBDR). There are other things like multi-pass (e.g. userland shadows, upsampling) or post-processing which are locked down by Meta which are endemic to a vertical system like this. Reminder I have a $1k bounty on this or #26160 since this is a massive uplift for WebXR as a platform.

[hybridherbst]

@CodyJasonBennett I understand that WebXR on WebGPU isn't even fully specified at this point.

My proposal (#28968 (comment)) is that the WebGL2 backend of the new WebGPURenderer could support WebXR today, and thus open the path towards actually using the "new" three, including the Nodes architecture.

For the time being, XR applications would then use new WebGPURenderer({forceWebGL: true}) to stay in WebGL2.

[toji]

The proposal @CodyJasonBennett linked is indeed the missing piece needed on the browser level to make this work. The good news on that front is that I'm scheduled to start work on that very soon! I'll keep you updated as progress is made.

[danrossi]

Is there WebXR for WebGPU yet ? From what I saw it was just theoreticals. So when launching to WebXR have to switch over to WebGL. If using the WebGPU nodes system, have one WebGPURenderer renderer that forces WebGL2 and another WebGPU and switch to the WebGL2 renderer for launching to WebXR ? I might run some tests of that.

[Mugen87]

No need to run tests since no backend of WebGPURenderer supports WebXR, yet.

[danrossi]

That is what I just said. I mean you can possibly have the gpu rendererer in non XR. And when going to XR use another gpu renderer with webgl2 backend enabled. That is what I meant testing out if it will work.

[Mugen87]

I have lately invested some time in this issue. Current progress is visible in the following branch however the implementation is not usable yet. https://github.com/Mugen87/three.js/commits/dev2/

One main issue in WebGPURenderer is that ArrayCamera is currently not functional because camera uniforms are not updated correctly.

Camera uniforms are maintained in the shared RENDER UBO which means they are updated only once per render call. That is not sufficient for XR where we render each render list more than once with sub cameras.

One obvious solution is to assign the TSL objects in accessors/Camera.js to the OBJECT group. A more optimized solution would be having UBOs per sub draw call however that requires a larger change in how ArrayCamera is processed. I'm just not happy with using the OBJECT group since that will affect all applications no matter if they use XR or not. Maybe we find a way to force the camera uniform update every time the renderer switches to a new sub camera.

If we go down the path and use the OBJECT group, we still need unique render objects for each eye. An obvious solution is to add the camera in this section:

three.js/src/renderers/common/RenderObjects.js

Lines 95 to 98 in 9ddf4be

	_chainKeys[ 0 ] = object;
	_chainKeys[ 1 ] = material;
	_chainKeys[ 2 ] = renderContext;
	_chainKeys[ 3 ] = lightsNode;

I initially though this bit isn't required however the current render context of the renderer is requested with the ArrayCamera. So when drawing objects with different sub cameras, you end up with the same render object and thus the camera uniforms won't be updated.

[hybridherbst]

Thanks for looking into this! Out of curiosity – maybe this is a good time for @cabanier to chime in regarding architectural choices for multiview in this new backend?

[Mugen87]

I'll continue to investigate different options by porting the following example to WebGPURenderer tomorrow so we have something for testing: https://threejs.org/examples/webgl_camera_array

This demo does currently not render correctly because of the mentioned reasons. Since it does not use the WebXR Device API, it's easier to focus on the uniform issue. If we manage to fix it, the new XRManager should eventually render correctly.

Any help in this topic is appreciated^^.

Side note: The existing code of XRManager is incomplete. I have just ported a single minimal code path without layers or WEBGL_multisampled_render_to_texture support. That can be implemented at a later point. I also want to add a proper JSDoc like with all new modules. The design goal of XRManager should be to move all WebGL related code as good as possible out of the manager into the backend. In this way, we can hopefully add WebGPU support a bit easier in the future.