EXRLoader: Added lossyDctChannelDecode

[mrdoob]

Related issue: #31429

Description

@sciecode Was curious if Claude Code could implement your comment. What do you think?

It creates a texture using THREE.RedFormat as texture.format and loads this:

With a custom shader that map the other channels it looks as it does in GIMP:

float luminance = texture2D( map, vUv ).r;
gl_FragColor = vec4( luminance, luminance, luminance, 1.0 );

Should EXRLoader return a RedFormat texture?
Or should it duplicate the values and alwasys return a RGBAFormat texture?

/cc @drone1 @WestLangley

[sciecode]

@sciecode Was curious if Claude Code could implement your comment. What do you think?

I'm not gonna lie, that's quite impressive.

At a quick glance everything looks good, there's one minor detail that isn't exactly correct:

// Handle multi-channel RGB sets
if ( cscSet.idx[ 0 ] !== undefined && channelData[ cscSet.idx[ 0 ] ] ) {
    lossyDctDecode( cscSet, rowOffsets, channelData, acBuffer, dcBuffer, outBuffer );
}

In theory this check doesn't handle some "possible" cases, like when the input file has multiple sets of RGB channels.

But in reality, I have yet to see a file that requires this, and the rest of the loader would also need to be reworked to output multiple textures instead of just one.

So, while not strictly correct, I think this is good enough for THREE.js needs.

Should EXRLoader return a RedFormat texture? Or should it duplicate the values and alwasys return a RGBAFormat texture?

We had this discussion at some point in the past, as to what should happen when the file outputs a single channel. IIRC, at the time, @Mugen87 recommended we default to outputting RedFormat. But, as you noticed, it is not automatically handled by our default shader library.

I ultimately have no say in this, so I defer the decision to you guys, but utilizing just a single channel texture does reduce the overall memory footprint by a huge margin. So it is the preferred option. Perhaps it would be possible to extend the shader chunks to handle single-channel case, or just keep things as they are, and require users to build a custom shader / use onBeforeCompile to adjust the texture sample on the fly.

[Mugen87]

The preference for RedFormat was discussed here: #20195 (comment)

[mrdoob]

Should we make our shaders treat RedFormat as luminance then?

People will be confused if they load a EXR in GIMP and Blender and shows white but it shows red in Three.js...

[Mugen87]

We can't check in the shader for RedFormat so this must be done before the shader is generated. However, there are use cases where developers specifically want to use a single or two channel texture so I would advice against an automatism that treats those formats as RGBA.

If we want to focus on easiness, I would recommend EXRLoader just returns a RGBA texture by default. We could add a loader flag called preserveColorChannels with false as default value. If set to true, the loader retains the color channels as defined in the EXR. How does that sound?

[mrdoob]

However, there are use cases where developers specifically want to use a single or two channel texture so I would advice against an automatism that treats those formats as RGBA.

Can you share some of those use cases? 🤔

[Mugen87]

Should we make our shaders treat RedFormat as luminance then?

Maybe I misunderstood this line.

Are you suggesting to not expanding the data to four channels in the loader but to "emulate" in some way how a single channel texture is sampled in the shader?

The use cases I was originally referring to are memory/bandwidth sensitive applications where expanding data to four channels is an unwanted operation.

[mrdoob]

Actually, I've asked Gemini about it and seems like WebGL 2 has a solution for this.

This is what it suggested:

// Assuming 'renderer' is your WebGL2Renderer and 'texture' is your RedFormat texture
const properties = renderer.properties.get(texture);

if (properties) {
    renderer.getContext().bindTexture(renderer.getContext().TEXTURE_2D, properties.__webglTexture);
    // Tell the sampler to use the RED channel for the R, G, and B components
    renderer.getContext().texParameteri(renderer.getContext().TEXTURE_2D, renderer.getContext().TEXTURE_SWIZZLE_G, renderer.getContext().RED);
    renderer.getContext().texParameteri(renderer.getContext().TEXTURE_2D, renderer.getContext().TEXTURE_SWIZZLE_B, renderer.getContext().RED);
    renderer.getContext().bindTexture(renderer.getContext().TEXTURE_2D, null);
}

I'm going to give it a try 👀

[Mugen87]

Um, I've never seen these enum values used with texParameterf ()/texParameteri(). According to the WebGL 1/2 spec, the following are valid:

TEXTURE_BASE_LEVEL
TEXTURE_COMPARE_FUNC
TEXTURE_COMPARE_MODE
TEXTURE_MAG_FILTER
TEXTURE_MAX_LEVEL
TEXTURE_MAX_LOD
TEXTURE_MIN_FILTER
TEXTURE_MIN_LOD
TEXTURE_WRAP_R
TEXTURE_WRAP_S
TEXTURE_WRAP_T

[mrdoob]

Indeed...

TEXTURE_SWIZZLE_G and TEXTURE_SWIZZLE_B are actually not available in WebGL...

I'll continue investigating...

[mrdoob]

I guess the only way would be to do this:

export default /* glsl */`
#ifdef USE_MAP

	vec4 sampledDiffuseColor = texture2D( map, vMapUv );

	#ifdef MAP_RED_AS_LUMINANCE
		sampledDiffuseColor.rgb = sampledDiffuseColor.rrr;
	#endif

	#ifdef DECODE_VIDEO_TEXTURE

		// use inline sRGB decode until browsers properly support SRGB8_ALPHA8 with video textures (#26516)

		sampledDiffuseColor = sRGBTransferEOTF( sampledDiffuseColor );

	#endif

	diffuseColor *= sampledDiffuseColor;

#endif
`;

And then in WebGLProgram we would need 23 more cacheKeyBooleans...

if ( parameters.mapRed )
	_programLayers.enable( 0 );
if ( parameters.alphaMapRed )
	_programLayers.enable( 1 );
if ( parameters.lightMapRed )
	_programLayers.enable( 2 );
if ( parameters.aoMapRed )
	_programLayers.enable( 3 );
if ( parameters.bumpMapRed )
	_programLayers.enable( 4 );
if ( parameters.normalMapRed )
	_programLayers.enable( 5 );
if ( parameters.displacementMapRed )
	_programLayers.enable( 6 );
if ( parameters.emissiveMapRed )
	_programLayers.enable( 7 );
if ( parameters.metalnessMapRed )
	_programLayers.enable( 8 );
if ( parameters.roughnessMapRed )
	_programLayers.enable( 9 );
if ( parameters.anisotropyMapRed )
	_programLayers.enable( 10 );
if ( parameters.clearcoatMapRed )
	_programLayers.enable( 11 );
if ( parameters.clearcoatNormalMapRed )
	_programLayers.enable( 12 );
if ( parameters.clearcoatRoughnessMapRed )
	_programLayers.enable( 13 );
if ( parameters.iridescenceMapRed )
	_programLayers.enable( 14 );
if ( parameters.iridescenceThicknessMapRed )
	_programLayers.enable( 15 );
if ( parameters.sheenColorMapRed )
	_programLayers.enable( 16 );
if ( parameters.sheenRoughnessMapRed )
	_programLayers.enable( 17 );
if ( parameters.specularMapRed )
	_programLayers.enable( 18 );
if ( parameters.specularColorMapRed )
	_programLayers.enable( 19 );
if ( parameters.specularIntensityMapRed )
	_programLayers.enable( 20 );
if ( parameters.transmissionMapRed )
	_programLayers.enable( 21 );
if ( parameters.thicknessMapRed )
	_programLayers.enable( 22 );

Definitely annoying, but maybe worth it?

[sciecode]

I think this boils down to a larger discussion. Which is how much it is expected of the library to just work without much in-depth knowledge of computer graphics.

These render optimizations are inherently unique to each project and use cases. I would say that for the vast majority of projects this reduction of memory footprint won't matter, cause their projects are just too simple. But these things do compound as projects grow in scale.

However, if you are working on an optimized project, you aren't even likely to use an EXR with just a single channel, nor use the default shaders.

You are probably gonna author your own texture with, say, different channels packed into a single block-compressed texture (e.g. metalness, ao, roughness, specular ), and you are also gonna modify the default shader library or create your own shader to handle your specific use-case.

Which is to say, any seasoned graphics developer will know how to deal with a RedFormat texture or even other variants (which I'm not even sure are currently supported, e.g RGFormats).

But not your average user, and I believe most of the library behaviour is geared towards making things simpler for newbies. So while handling this at a shader generation is annoying, it is likely the best way to make things just work by default.

[Mugen87]

I was hoping to not patch the shaders in that way 🙈. The implementation will end up quite messy in WebGLRenderer. I would rather vote to enhance EXRLoader and expand single-channel data to four channels. But this should be controlled with a boolean like suggested above.

Interpreting texture samples as luminance values will be easier to support with TSL. I've hacked around a bit and there are multiple places where we can support this feature for all color textures (not depth textures of course). We could add an extension to the snippet variable below this line:

three.js/src/nodes/accessors/TextureNode.js

Line 475 in e4139cf

const snippet = this.generateSnippet( builder, textureProperty, uvSnippet, levelSnippet, biasSnippet, depthSnippet, compareSnippet, gradSnippet );

Or we enhance the node builder and implement something in NodeBuilder.format(). I bet @sunag can recommend an appropriate place for such a logic.

I've tested this approach with webgpu_sandbox by changing the data-texture to a single-channel texture and then controlling the behavior with a new texture boolean luminance. This boolean essentially means whether samples should be treated as luminance values or not (default is false).

[mrdoob]

@sciecode

However, if you are working on an optimized project, you aren't even likely to use an EXR with just a single channel, nor use the default shaders.

That's true. We can just copy the values to the other channels and return RGBAFormat then.

We can revisit RedFormat issue once we bump into a more compelling use case.

[mrdoob]

Okay, I'll merge this now and I'll do another PR with the RGBAFormat change.

[sciecode]

Okay, I'll merge this now and I'll do another PR with the RGBAFormat change.

Should be as simple as removing outputChannels from this conditional and always setting it as 4. (Keep the decodeChannels inside the conditional).

three.js/examples/jsm/loaders/EXRLoader.js

Lines 2466 to 2476 in f9591c4

	if ( channels.R && channels.G && channels.B ) {
	fillAlpha = ! channels.A;
	EXRDecoder.outputChannels = 4;
	EXRDecoder.decodeChannels = { R: 0, G: 1, B: 2, A: 3 };
	} else if ( channels.Y ) {
	EXRDecoder.outputChannels = 1;
	EXRDecoder.decodeChannels = { Y: 0 };

Remove this.

three.js/examples/jsm/loaders/EXRLoader.js

Lines 2577 to 2582 in f9591c4

	} else {
	EXRDecoder.format = RedFormat;
	EXRDecoder.colorSpace = NoColorSpace;
	}

And after decoding is done, if (info.inputChannels.length < 3) copy R-channel to GB and setting A to 1 in EXRDecoder.byteArray. Which should already be in the familiar RGBA-RGBA-RG... organization.

three.js/examples/jsm/loaders/EXRLoader.js

Line 2632 in f9591c4

EXRDecoder.decode();

PS: if EXRDecoder.byteArray is Uint16Array, it means the data is HalfFloatType, so set the alpha to 0x3C00 which is the encoded alpha 1 in half precision float.

I can probably manage this PR over the weekend, if you prefer.

Edit: #31511

[brennercruvinel]

he's alive <3