Heritage-NeRF: Neural Radiance Fields for 3D Heritage Reconstruction

A PyTorch-based implementation of Neural Radiance Fields (NeRF) optimized for high-fidelity 3D reconstruction of cultural heritage sites. This project integrates COLMAP Structure-from-Motion data to address the challenges of sparse and noisy multi-view documentation.

📦 Data Setup (Required)

The 8.5 GB dataset used in this project is excluded from the GitHub repository to maintain a lightweight codebase.

• Download: https://huggingface.co/datasets/Manognaaaaaa/Heritage-NeRF-Data
• Setup: Extract the data.zip file into the data/ folder in your root directory.

2. Directory Structure: Create a data/ folder in the root directory and extract the files so they match the structure below:

   data/nerfbk/
   ├── heritage/        # Input RGB images
   └── sparse/0/        # COLMAP binary files (cameras.bin, images.bin, points3D.bin)
   

🚀 Installation & Quick Start

Clone the repository:

git clone https://github.com/your-username/Heritage-NeRF.git
cd Heritage-NeRF

Create local directories: Since these are ignored by Git, you must create them for the code to output results:

mkdir results
mkdir data

Set up Environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt

Run the Pipeline: Open Code_main.ipynb in Jupyter and run all cells.

🛠️ Project Structure

├── Code_main.ipynb          # Main Jupyter notebook with full pipeline
├── data/                    # (Local only) Project datasets
├── results/                 # (Local only) Model outputs and metrics
├── .gitignore               # Prevents large data uploads
└── README.md

🌟 Key Technical Features

• COLMAP Binary Parsing: Automatic handling of cameras.bin, images.bin, and points3D.bin
• Heritage-NeRF Model: Enhanced version addressing sparse/noisy view challenges with cross-view color consistency
• Advanced Regularization: Includes reflection-aware normal smoothness and occupancy penalties
• Multi-Metric Evaluation: Quantifies results using PSNR, SSIM, and Chamfer distance

💻 Performance & Hardware

• Optimized for Consumer GPUs: This project was developed and tested on an HP Victus 15
• VRAM Requirements: 8GB+ recommended for batch_size=1024. If you encounter "CUDA Out of Memory," reduce the batch size in the Configuration cell

Running the Pipeline

Open and run Code_main.ipynb in Jupyter Notebook or JupyterLab:

jupyter notebook Code_main.ipynb

Configuration

All hyperparameters are defined in the notebook's Configuration Parameters section (Cell #2):

# Model parameters
pos_encoding_L = 10        # Positional encoding levels
hidden_dim = 256           # MLP hidden dimension
num_layers = 8             # Number of MLP layers

# Training parameters
num_iterations = 3000      # Training iterations
batch_size = 1024          # Rays per batch
learning_rate = 5e-4       # Learning rate

# Loss weights
w_img = 1.0               # Image reconstruction weight
w_ssim = 0.1              # SSIM loss weight
w_norm = 0.01             # Normal smoothness weight
w_consistency = 0.05      # Cross-view consistency

Switching Between Modes

• Baseline Mode: For dense multi-view scenarios

  baseline_mode = True

• Heritage-NeRF Mode: For sparse/noisy views (default)

  baseline_mode = False

Data Format

Expected Input Structure

data/nerfbk/
├── heritage/
│   ├── frame_0.jpg
│   ├── frame_1.jpg
│   └── ... (RGB images)
└── sparse/0/
    ├── cameras.bin
    ├── images.bin
    └── points3D.bin

COLMAP Workflow

1. Run COLMAP structure-from-motion on heritage images
2. Export sparse reconstruction to binary format
3. Place in data/nerfbk/sparse/0/

Output and Results

Results are saved to the results/ directory, including:

• metrics.csv: Quantitative evaluation metrics (PSNR, SSIM, Chamfer distance, F-score)
• Rendered images: Novel view synthesis results
• Geometry visualization: 3D mesh and point cloud visualizations
• Loss curves: Training convergence plots

Performance Notes

• GPU Memory: Recommended 8GB+ VRAM for batch_size=1024
• Training Time: ~2-4 hours on RTX 3090 for 3000 iterations
• Resolution: Default batch_size assumes 512×512 or smaller images

Architecture Details

Positional Encoding

PE(p) = [sin(2^0 * p), cos(2^0 * p), ..., sin(2^9 * p), cos(2^9 * p)]

NeRF MLP

Input: [PE(position), PE(direction)]
  → 8 fully connected layers (256 dims)
  → Output: [RGB, Density]

Volume Rendering

C(r) = ∫ T(t) · σ(t) · c(t) dt

where T(t) = exp(-∫ σ(s) ds), σ is density, c is color

References

• NeRF Paper: Mildenhall et al., "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis", ECCV 2020
• COLMAP: Schönberger & Frahm, "Structure-from-Motion Revisited", CVPR 2016

Troubleshooting

CUDA Out of Memory

• Reduce batch_size in configuration
• Reduce image resolution
• Use baseline_mode = True

Poor Reconstruction Quality

• Increase num_iterations
• Adjust loss weights (especially w_consistency for sparse views)
• Ensure COLMAP reconstruction quality is adequate

Missing COLMAP Files

• Verify COLMAP binary files are in data/nerfbk/sparse/0/
• Check file permissions and disk space

Future Enhancements

• Real-time rendering with WebGL
• Multi-resolution mipmap pyramid
• Appearance flow modeling
• Dynamic scene support
• Interactive refinement UI

Data Setup

The 8.5 GB dataset used in this project is excluded from the repository to maintain a lightweight codebase.

Download: [Insert your Google Drive or OneDrive link here].

Placement: Extract the files into the data/ folder following the structure defined in the Data Format section.