From 4ed55a2fdf65df4c9548c195c12fb870695a6615 Mon Sep 17 00:00:00 2001 From: factral Date: Fri, 18 Jul 2025 13:10:55 -0400 Subject: [PATCH 1/2] added unmin-nerf --- docs/index.md | 1 + docs/nerfology/methods/index.md | 1 + docs/nerfology/methods/unmix_nerf.md | 175 +++++++++++++++++++++++++ nerfstudio/configs/external_methods.py | 15 +++ 4 files changed, 192 insertions(+) create mode 100644 docs/nerfology/methods/unmix_nerf.md diff --git a/docs/index.md b/docs/index.md index f43ec385a1..5a58e33f9f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -148,6 +148,7 @@ This documentation is organized into 3 parts: ### Third-party Methods +- [UnMix-NeRF](nerfology/methods/unmix_nerf.md): Spectral-Unmixing Meets Neural Radiance Fields - [BioNeRF](nerfology/methods/bionerf.md): Biologically Plausible Neural Radiance Fields for View Synthesis - [Instruct-NeRF2NeRF](nerfology/methods/in2n.md): Editing 3D Scenes with Instructions - [Instruct-GS2GS](nerfology/methods/igs2gs.md): Editing 3DGS Scenes with Instructions diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md index 320d6ae97f..a84ed37b16 100644 --- a/docs/nerfology/methods/index.md +++ b/docs/nerfology/methods/index.md @@ -49,6 +49,7 @@ The following methods are supported in nerfstudio: BioNeRF NeRFtoGSandBack OpenNeRF + UnMix-NeRF ``` (own_method_docs)= diff --git a/docs/nerfology/methods/unmix_nerf.md b/docs/nerfology/methods/unmix_nerf.md new file mode 100644 index 0000000000..a94c884e6b --- /dev/null +++ b/docs/nerfology/methods/unmix_nerf.md @@ -0,0 +1,175 @@ +# UnMix-NeRF + +

Spectral Unmixing Meets Neural Radiance Fields

+ +```{button-link} https://www.arxiv.org/pdf/2506.21884 +:color: primary +:outline: +Paper +``` + +```{button-link} https://www.factral.co/UnMix-NeRF/ +:color: primary +:outline: +Project Page +``` + +
+ +TL;DR _We propose UnMix-NeRF, the first method integrating spectral unmixing into NeRF, enabling hyperspectral view synthesis, accurate unsupervised material segmentation, and intuitive material-based scene editing, significantly outperforming existing methods._ + +_ICCV 2025_ + +UnMix-NeRF Overview
+ +
+ +## Visual Results + +

Hotdog Scene

+ + + + + + +
+ +

RGB

+ 		+ +

Unsupervised Material Segmentation

+ 		+ +

Scene Editing

+
+ +

Ajar Scene

+ + + + + + +
+ +

RGB

+ 		+ +

Unsupervised Material Segmentation

+ 		+ +

PCA Visualization

+
+ +## Installation + +Install nerfstudio dependencies following the [installation guide](https://docs.nerf.studio/quickstart/installation.html). + +Then install UnMix-NeRF: + +```bash +git clone https://github.com/Factral/UnMix-NeRF +cd UnMix-NeRF +pip install -r requirements.txt +pip install . +``` + +## Running UnMix-NeRF + +Basic training command: + +```bash +ns-train unmixnerf \ + --data \ + --pipeline.num_classes \ + --pipeline.model.spectral_loss_weight 5.0 \ + --pipeline.model.temperature 0.4 \ + --experiment-name my_experiment +``` + +## Method + +### Overview + +[UnMix-NeRF](https://www.arxiv.org/pdf/2506.21884) Neural Radiance Field (NeRF)-based segmentation methods focus on object semantics and rely solely on RGB data, lacking intrinsic material properties. This limitation restricts accurate material perception, which is crucial for robotics, augmented reality, simulation, and other applications. We introduce UnMix-NeRF, a framework that integrates spectral unmixing into NeRF, enabling joint hyperspectral novel view synthesis and unsupervised material segmentation. + +Our method models spectral reflectance via diffuse and specular components, where a learned dictionary of global endmembers represents pure material signatures, and per-point abundances capture their distribution. For material segmentation, we use spectral signature predictions along learned endmembers, allowing unsupervised material clustering. Additionally, UnMix-NeRF enables scene editing by modifying learned endmember dictionaries for flexible material-based appearance manipulation. Extensive experiments validate our approach, demonstrating superior spectral reconstruction and material segmentation to existing methods. + +### Pipeline + +![UnMix-NeRF Pipeline](https://www.factral.co/UnMix-NeRF/assets/framework.jpg)
+ +## Data Format + +UnMix-NeRF extends standard nerfstudio data conventions to support hyperspectral data: + +### Required Structure + +``` +data/ +├── transforms.json # Camera poses (standard) +├── images/ # RGB images (standard) +│ ├── frame_00001.jpg +│ └── ... +├── hyperspectral/ # Hyperspectral data (NEW) +│ ├── frame_00001.npy # Shape: (H, W, B) +│ └── ... +└── segmentation/ # Ground truth (optional) + ├── frame_00001.png + └── ... +``` + +### Hyperspectral Data + +- **Format**: `.npy` files with dimensions `(H, W, B)` +- **Values**: Normalized between 0 and 1 +- **Bands**: Number of spectral channels (B) + +Update your `transforms.json`: + +```json +{ + "frames": [ + { + "file_path": "./images/frame_00001.jpg", + "hyperspectral_file_path": "./hyperspectral/frame_00001.npy", + "seg_file_path": "./segmentation/frame_00001.png", + "transform_matrix": [...] + } + ] +} +``` + +## Key Parameters + +| Parameter | Description | Default | +| --------------------------------------- | --------------------------------- | ------- | +| `--pipeline.num_classes` | Number of material endmembers | 6 | +| `--pipeline.model.spectral_loss_weight` | Weight for spectral loss | 5.0 | +| `--pipeline.model.temperature` | Temperature for abundance softmax | 0.4 | +| `--pipeline.model.load_vca` | Initialize with VCA endmembers | False | +| `--pipeline.model.pred_specular` | Enable specular component | True | + +## Citation + +```bibtex +@inproceedings{perez2025unmix, + title={UnMix-NeRF: Spectral Unmixing Meets Neural Radiance Fields}, + author={Perez, Fabian and Rojas, Sara and Hinojosa, Carlos and Rueda-Chac{\'o}n, Hoover and Ghanem, Bernard}, + booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision}, + year={2025} +} +``` diff --git a/nerfstudio/configs/external_methods.py b/nerfstudio/configs/external_methods.py index 002b3299b6..148323334a 100644 --- a/nerfstudio/configs/external_methods.py +++ b/nerfstudio/configs/external_methods.py @@ -298,6 +298,21 @@ class ExternalMethod: ) ) +# UnMix-NeRF +external_methods.append( + ExternalMethod( + """[bold yellow]UnMix-NeRF[/bold yellow] +For more information visit: https://docs.nerf.studio/nerfology/methods/unmix_nerf.html + +To enable UnMix-NeRF, you must install it first by running: + [grey]pip install git+https://github.com/Factral/UnMix-NeRF[/grey]""", + configurations=[ + ("unmix-nerf", "UnMix-NeRF, used in paper"), + ], + pip_package="git+https://github.com/Factral/UnMix-NeRF", + ) +) + @dataclass class ExternalMethodDummyTrainerConfig: From 60fef76718f691d4dadf2552cee6cd5b7dd3adaa Mon Sep 17 00:00:00 2001 From: factral Date: Fri, 18 Jul 2025 13:17:20 -0400 Subject: [PATCH 2/2] fix typo --- docs/nerfology/methods/unmix_nerf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/nerfology/methods/unmix_nerf.md b/docs/nerfology/methods/unmix_nerf.md index a94c884e6b..a6fb65867e 100644 --- a/docs/nerfology/methods/unmix_nerf.md +++ b/docs/nerfology/methods/unmix_nerf.md @@ -140,7 +140,7 @@ data/ Update your `transforms.json`: -```json +```text { "frames": [ {