KHR_gaussian_splatting proposal spec#72
KHR_gaussian_splatting proposal spec#72keyboardspecialist wants to merge 4 commits intocesium-nativefrom
Conversation
KHR_gaussian_splatting proposal spec
|
This looks fine to me @keyboardspecialist, and I'm happy to see it merged into the |
lilleyse
left a comment
lilleyse
left a comment
There was a problem hiding this comment.
@keyboardspecialist this looks pretty good. For now I just have comments on the technical aspects, but I think this could also use a style pass.
| ### Extension attributes | ||
|
|
||
| `extensions.KHR_gaussian_splatting` may contain the following values: | ||
|
|
||
| | Attributes | Type | Description | Required | | ||
| | --- | --- | --- | --- | | ||
| | quantizedPositionScale | number | Scale value for dequantizing POSITION attribute values | No, default: `1.0` | | ||
|
|
||
| ```json | ||
| { | ||
| "extensions": { | ||
| "KHR_gaussian_splatting": { | ||
| "quantizedPositionScale": 13.228187255859375 | ||
| } | ||
| } | ||
| } | ||
| ``` |
There was a problem hiding this comment.
Is quantizedPositionScale necessary? Usually with KHR_mesh_quantization the quantization scale is stored in the node transform.
| | Splat Data | glTF Attribute | Accessor Type | Component Type | | ||
| | --- | --- | --- | --- | | ||
| | Position | POSITION | VEC3 | float | | ||
| | Color (Spherical Harmonic 0 (Diffuse) and opacity) | COLOR_0 | VEC4 | unsigned byte normalized | | ||
| | Rotation | _ROTATION | VEC4 | float | | ||
| | Scale | _SCALE | VEC3 | float | |
There was a problem hiding this comment.
This table could be formatted more like the table in EXT_mesh_gpu_instancing
| | Splat Data | glTF Attribute | Accessor Type | Component Type | | ||
| | --- | --- | --- | --- | | ||
| | Position | POSITION | VEC3 | float | | ||
| | Color (Spherical Harmonic 0 (Diffuse) and opacity) | COLOR_0 | VEC4 | unsigned byte normalized | |
There was a problem hiding this comment.
Does COLOR_0 need to be restricted to VEC4 UNSIGNED_BYTE (normalized)? Someone may have VEC3 colors (no alpha) or floating point values. I feel like all the type/componentType combinations that glTF supports for vertex colors should be allowed here.
There was a problem hiding this comment.
Allowing float wouldn't be an issue I don't think. I don't know about omitting alpha. It's a core component of splatting. I would need to see the use case.
There was a problem hiding this comment.
True... I'm not sure if there's a use case for opaque splats.
There was a problem hiding this comment.
Given the conversations we have had with Adobe recently, I agree with @keyboardspecialist. I don't think that one could reasonably omit the alpha channel for splats. While we refer to the lowest-order spherical harmonic as diffuse color, that's not really what is going on there. The real view-dependent color is actually packed with the specular in the entire set of the spherical harmonics, and in that case the opacity is required.
Also, I just want to add that based on the conversations we've had with the community and Adobe, I'm guessing that fully supporting the higher-order spherical harmonics is going to need to be a major piece of our next steps in next year.
| } | ||
| ``` | ||
|
|
||
| ### Transforming Gaussian Splat Data for glTF |
There was a problem hiding this comment.
This section should be marked as non-normative.
|
|
||
| ## Adding Gaussian Splats to Primitives | ||
|
|
||
| As Gaussian splats are defined by a position color, rotation and scale, we both map to existing attributes and define new ones. These are attached to a primitive by defining the `extensions.KHR_gaussian_splatting` property on that mesh. |
There was a problem hiding this comment.
| As Gaussian splats are defined by a position color, rotation and scale, we both map to existing attributes and define new ones. These are attached to a primitive by defining the `extensions.KHR_gaussian_splatting` property on that mesh. | |
| As Gaussian splats are defined by a position color, rotation and scale, we both map to existing attributes and define new ones. These are attached to a primitive by defining the `extensions.KHR_gaussian_splatting` property on that primitive. |
|
|
||
| #### Sample | ||
|
|
||
| Basic example shown below. This sample shows adding Guassian splats to the first primitive of a mesh. No |
|
|
||
| ### Position Attribute | ||
|
|
||
| It is recommended to quantize to `half float` utilizing `KHR_mesh_quantization` with an `unsigned int` accessor type. If using scaled quantization values, you may set the `quantizedPositionScale` value in `extensions.KHR_gaussian_splatting`. |
There was a problem hiding this comment.
KHR_mesh_quantization doesn't support half float... and unsigned int doesn't provide any space savings. Wouldn't unsigned short or unsigned byte be more appropriate depending on your target quantization bits?
There was a problem hiding this comment.
You're correct. It's actually currently using unsigned short.
| | meshopt configuration | value | | ||
| | --- | --- | | ||
| | Mode | Attributes | | ||
| | Filter | None | |
There was a problem hiding this comment.
The glTF keys/values should probably be used in these tables, e.g.
| | meshopt configuration | value | | |
| | --- | --- | | |
| | Mode | Attributes | | |
| | Filter | None | | |
| | meshopt configuration | value | | |
| | --- | --- | | |
| | `mode` | `"ATTRIBUTES"` | | |
| | `filter` | `"NONE"` | |
|
Superseded by #81 |
4 participants
Clean branch with just KHR_gaussian_splatting proposal spec
Extension allows glTF files to treat point primitives as Gaussian splats.