WebGPURenderer: Images with color-specific metadata produces different results in both backends.

[steve-o]

Description

REF: https://www.vantajs.com/?effect=clouds2

Taking the popular Vanta clouds2 demo and porting to TSL I can reproduce the visual output with forceWebGL: true, however when using WebGPU the clouds are significantly less fluffy. I really not sure how to start debugging this.

Reproduction steps

1. In one window: https://miffy.dsbunny.com/apps/webgpu/play/webgl.html
2. In another: https://miffy.dsbunny.com/apps/webgpu/play/webgpu.html

Code

const cloudShader = /*@__PURE__*/ Fn( ({
	resolution,
	time,
	speed,
	skyColor,
	cloudColor,
	lightColor,
	cloudTexture,
}) => {
	const coord = vec2(
		screenCoordinate.x,
		screenCoordinate.y,
	).toVar();

	// Base direction vector (0.8, 0, coord/resolution.y - 0.35)
	const d = vec4(
		float(0.8),
		float(0),
		coord.div(resolution.y).sub(float(0.35)),
	).toVar();

	// Sky gradient based on height (skyColor - d.w)
	let outColor = skyColor.sub(d.w).toVar();

	// Initial value for ray marching (200.0 + sin(dot(coord,coord)))
	let t = float(200.0).add(sin(dot(coord, coord))).toVar();

	// Ray marching loop
	const MAX_ITER = float(100.0);

	Loop({ start: float(1.0), end: MAX_ITER, type: 'float', condition: '<=' }, () => {
		// Decrease t by 2.0 for each step
		t.assign(t.sub(float(2.0)));

		// Break if t < 0
		If(t.lessThan(float(0.0)), () => { Break(); });

		// p = 0.05 * t * d
		let p = d.mul(t).mul(float(0.05)).toVar();

		// Apply movement through space
    		// p.xz += time * 0.5 * speed
		p.xz.assign(p.xz.add(time.mul(float(0.5)).mul(speed)));

		// p.x += sin(time * 0.25 * speed) * 0.25
    		p.x.assign(p.x.add(sin(time.mul(float(0.25)).mul(speed)).mul(float(0.25))));

		// Start with s = 2.0
		let s = float(2.0).toVar();

		// Initialize f value for cloud density
    		// Note: In GLSL we have p.w + 1.0 - T - T - T - T
    		let f = p.w.add(float(1.0)).toVar();

    		// Emulate -T -T -T -T
		Loop(4, () => {
			// Calculate texture coordinates for sampling
      			// fract((s*p.zw + ceil(s*p.x)) / 200.0)
			const p_zw = p.zw;
			const s_p_zw = s.mul(p_zw);
			const s_p_x = s.mul(p.x);
			const texCoord = fract(s_p_zw.add(ceil(s_p_x)).div(float(200.0)));

			const sampleY = texture(cloudTexture, texCoord).y.div(s.assign(s.add(s)));
			const T = sampleY.mul(float(4.0));

			// Subtract T from f (f -= T)
      			f.assign(f.sub(T));
		});

		// Check if inside a cloud (f < 0.0)
		If(f.lessThan(float(0.0)), () => {
			// Calculate cloud color with shading
      			// mix(lightColor, cloudColor, -f)
			const shaded = mix(lightColor, cloudColor, f.negate());

			// Blend with the sky color
      			// mix(out1, cloudColorShading, -f * 0.4)
			const mixed = mix(outColor, shaded, f.negate().mul(float(0.4)));
			outColor.assign(mixed);
		});
	});

	// Return final color with alpha = 1.0
	return vec4(outColor, float(1.0));
} );

Live example

• jsfiddle-latest-release WebGPURenderer

Screenshots

Version

176

Device

Desktop

Browser

Chrome

OS

Windows

[Mugen87]

Updated live example: https://jsfiddle.net/rw2bnjvq/

[Mugen87]

The root cause seems to be the transparent PNG noise texture which is sampled differently in both backends. Using a JPG without transparency and both backends render the same: https://jsfiddle.net/7quds5nk/

Below are two live examples that isolate the issue. The code samples the noise texture (PNG) and displays the green value as a grayscale with constant alpha (1).

WebGPU: https://jsfiddle.net/c5tdLm63/
WebGL: https://jsfiddle.net/c5tdLm63/1/

The WebGL backend renders as expected like WebGLRenderer so the issue must be located somewhere in the WebGPU backend.

[Mugen87]

In Safari with enabled WebGPU, both fiddles render identical. Only in Chrome there is a difference.

Could this be a browser issue and Chrome's WebGPU implementation does not sample the semi-transparent color values correctly?

[steve-o]

Can raise with the Chome team and get their feedback? Thanks for updating the fiddles.

Chromium ticket: https://issues.chromium.org/issues/418746324

[Mugen87]

@greggman Do you think we see a potential Chromium issue here or do we overlook something WebGPU specific?

The issue is summed up in #31132 (comment). I've made sure the reproduction test cases do not apply any color transformation in the shaders (so no tone mapping or color space conversion).

[WestLangley]

This appears to be related to premultiplied alpha. These two test cases match on chrome (only).

WebGL: https://jsfiddle.net/3m7js4ry/

material.colorNode = vec4( color.r.mul( color.a ), color.g.mul( color.a ), color.b.mul( color.a ), 1 );

//

WebGPU: https://jsfiddle.net/9fjn4d2h/

material.colorNode = vec4( color.r, color.g, color.b, 1 );

[steve-o]

I think the Chromium ticket update is saying invalid API usage.

[greggman]

I think the Chromium ticket update is saying invalid API usage.

I don't think that's the case yet. Still checking...

[greggman]

I guess the short version for this exact problem is, replace that noise.png with a new one. The issue seems to be that noise.png has gAMA and cHRM chunks. Those are applied or ignored differently in WebGL vs WebGPU, partly as specced. IIUC, the workaround to ignore those chunks when using WebGPU is to load the image via createImageBitmap and pass in { colorSpaceConversion: 'none' }. Unfortunately, that path in all browsers still appears to apply at least one of those chunks differently in WebGPU vs WebGL so it's not currently a solution. Hopefully that can be fixed.

Another solution would be to decode the images yourself 🤮 Then you can chose to apply or not apply color space. I'm not recommending it but, mentioning it since getting browsers to handle this stuff consistently is difficult. The good thing is deflate decompression is now JavaScript API so it might not be that much code to write a PNG loader.

ps: still need to find why things are different in Safari for the examples above

[Mugen87]

Thank you for this valuable feedback! It's somewhat a relief the issue is so good understood and it is not a bug in the WebGPU implementation or on our side.

It's actually not the first time that such metadata result in unexpected behavior. For example #30471 demonstrates how browsers evaluate rotation metadata in MPEG files in different ways leading to inconsistent behavior of VideoTexture. The best solution so far is to encode the MPEG without such metadata.

It's true that with custom image loaders the engine would have more control over the image data. On the other side this would add additional maintenance burdens. Besides, developers can load image data without one of our loaders (ImageLoader, ImageBitmapLoader or TextureLoader) and then we have the same issue again. I personally would prefer to rely on the Web API for such tasks.

Regarding the discussion at the Chromium bug: I like the suggestion of a consistent behavior of createImageBitmap() across WebGL and WebGPU. In the long term ImageBitmapLoader becomes hopefully the default image loader in the engine. In context of glTF, the respective importer already uses it by default when support is detected. Next to the improved runtime behavior, are more consistent result when processing images would be an additional argument for using image bitmaps.

[greggman]

Ok, I narrowed down the Safari issue. Safari has a bug where you get different results from these 2 paths

const img = new Image();
img.src = 'noise.png'
await img.decode();
device.queue.copyExternalImageToTexture(...)

vs

const img = new Image();
img.src = 'noise.png'
img.addEventListener('load', () => {
  device.queue.copyExternalImageToTexture(...)
});

• use decode
• use addEventListener

https://bugs.webkit.org/show_bug.cgi?id=293284

[Mugen87]

FYI: I have "fixed" the problematic noise texture via #31137. Since the Chromium bug refers to https://threejs.org/examples/textures/noise.png, the link might be outdated with the next release r177 next week since the new image produces consistent results.

[greggman]

No worries, I copied the .png here

https://greggman.github.io/doodles/images/noise-with-gama-and-chrm-chunks.png

[greggman]

Just for fun I asked Gemini 2.5 Pro (preview) to convert that python PNG loader into JavaScript. It worked 😱

I'm not suggesting you use it. Just 😱😱😱

[mrdoob]

That's way simpler than the one we used before.

[greggman]

Doh!, it looks like using createImageBitmap can be used with the original noise.png file

    const res = await fetch(url);
    const blob = await res.blob();
    const bitmap = await createImageBitmap(blob, {
      colorSpaceConversion: 'none',   // important - default is implementation defined 🤷‍♂️
      premultiplyAlpha: 'none',       // important - default is implementation defined 🤷‍♂️
    });

I missed that premultiplyAlpha: 'none' was needed 😅

It gets more confusing though. In WebGL it's specified as follows

This color space conversion applies to ImageBitmap objects as well, though other texture unpack parameters do not apply to ImageBitmaps because they are expected to be specified during ImageBitmap construction. Implementation experience revealed that it was beneficial to perform ImageBitmaps' color space conversion as late as possible when uploading to WebGL textures.

In other words gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, ???) is ignored for ImageBitmap so that usage in three.js would have to take that into account.

WebGPU doesn't have this exception for ImageBitmap

[Mugen87]

Instances of ImageBitmapLoader have the premultiplyAlpha option disabled by default. This setting is usually not changed in code using ImageBitmapLoader.

three.js/src/loaders/ImageBitmapLoader.js

Line 67 in 6c8e828

this.options = { premultiplyAlpha: 'none' };

Besides, the texture property that controls the UNPACK_PREMULTIPLY_ALPHA_WEBGL flag is also set to false by default.

three.js/src/textures/Texture.js

Lines 258 to 268 in 6c8e828

	/**
	* If set to `true`, the alpha channel, if present, is multiplied into the
	* color channels when the texture is uploaded to the GPU.
	*
	* Note that this property has no effect when using `ImageBitmap`. You need to
	* configure premultiply alpha on bitmap creation instead.
	*
	* @type {boolean}
	* @default false
	*/
	this.premultiplyAlpha = false;

So if the user doesn't change anything, image bitmaps and normal images should be treated identically.