PRENOM uses category-level meta-learned NeRF priors to accelerate object reconstruction and enable 4-DoF canoical object pose estimation. Priors are trained by Reptile meta-learning on numerous synthetic reconstruction tasks, while the NeRF architecture is also optimized per category using a multi-objective mixed-variable genetic algorithm. The system architecture can be seen below:
BASELINE is an RGB-D version of RO-MAP.
- Eigen3 (test version: 3.4.0)
- Pangolin (test version: v0.9)
- CMake (test version: 3.22.1)
- PCL (test version 1.10)
Tested on:
- Ubuntu 20.04 LTS, GPU: T600 Laptop GPU (Nvidia driver: 535), CUDA 12.2, OpenCV 4.2.0
- Ubuntu 24.04 LTS, GPU: RTX 5090 (Nvidia driver: 570), CUDA 12.8, OpenCV 4.6.0
Clone the repository and it's dependencies
git clone --recursive git@github.com:snt-arg/PRENOM.git
Similar to RO-MAP, build first the Multi-Object-NeRF module and then the SLAM system
cd dependencies/Multi-Object-NeRF
sh build.sh <number-of-cores>
cd ../../
sh build.sh <number-of-cores>
The six synthetic sequences (from S0 to S5 in the paper) and the two real sequences (scene1 and scene2 from RO-MAP) can be downloaded from here. S0 corresponds to the Cube Diorama Dataset, albiet with config modified to work with PRENOM. The real sequences are the ones provided by RO-MAP but also with the configuration modified.
The priors for 9 categories are available in the cookbook folder, with their YOLO class IDs as the folder name.
| Category name | YOLO category name | ID |
|---|---|---|
| ball | sports ball | 32 |
| mug | cup | 41 |
| chair | chair | 56 |
| plant | potted plant | 58 |
| display | tv | 62 |
| laptop | laptop | 63 |
| mouse | mouse | 64 |
| keyboard | keyboard | 66 |
| book | book | 73 |
Each folder consists of:
weights.json- the meta-learned initial NeRF parameters.model.ply- the normalized mesh used to align the prior with the actual object.density.ply- the prior density grid used to align priors and for probabilistic ray sampling.network.json- the category-specific architecture that was optimised.config.json- category-specific object association configuration.
The default configurations can be found in cookbook/0/, and cookbook/recipes.txt lists all the known categories along with their class IDs.
Details on how to train a new prior are in the README of the learntolearn folder.
To run, navigate to the PRENOM directory and run using:
__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia ./examples/RGBD/rgbd_tum ./vocabulary/ORBvoc.bin ./dependencies/Multi-Object-NeRF/Core/configs/base.json <path/to/dataset/S1>
You can also run a monocular version by using the target ./examples/Monocular/mono_tum. However, monocular data association is inherently noisy and the run has to be repeated until satisfactory results, as also mentioned by RO-MAP.
struct cudaPointerAttributes has no member named 'memoryType'while building PRENOM.
This can happen since we are using an older version of Pangolin, and in later versions of CUDA, this attribute was renamed totype. To fix this, just rename.memoryTypeto.typein the file/usr/local/include/pangolin/image/memcpy.h(or equivalent location of your Pangolin installation) and re-build PRENOM.
This repo is built on top of the RO-MAP framework, and borrows a significant amount of their codebase. We thank the authors for their work. Other acknowledgements to:
- SLAM: ORB-SLAM2, CubeSLAM, EAO-SLAM
- NeRF: instant-ngp
- Datasets: Cube Diorama Dataset
- Farthest Point Sampling: FPS
This repo is GPLv3 Licensed (inherit ORB-SLAM2 and RO-MAP). The Multi-Object NeRF system based on tiny-cuda-nn (BSD 3-clause license). The implementation refers to instant-ngp (Nvidia Source Code License-NC).
If you found this code/work useful, consider citing our work:
@INPROCEEDINGS{11247175,
author={Ejaz, Saad and Bavle, Hriday and Ribeiro, Laura and Voos, Holger and Sanchez-Lopez, Jose Luis},
booktitle={2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
title={Category-level Meta-learned NeRF Priors for Efficient Object Mapping},
year={2025},
volume={},
number={},
pages={9186-9193},
doi={10.1109/IROS60139.2025.11247175}
}
Additionally, consider citing RO-MAP that this work builds upon:
@ARTICLE{RO-MAP,
author={Han, Xiao and Liu, Houxuan and Ding, Yunchao and Yang, Lu},
journal={IEEE Robotics and Automation Letters},
title={RO-MAP: Real-Time Multi-Object Mapping With Neural Radiance Fields},
year={2023},
volume={8},
number={9},
pages={5950-5957},
doi={10.1109/LRA.2023.3302176
}