Deformable Instance MetalSplatter

This project is forked from an amazing work MetalSplatter. It provides support for running deformable splats as well as instance segmented splats.

Render deformable 3D Gaussian Splats using Metal on Apple platforms. Tested on :

• Iphone 15, Iphone 17 Air
• Ipad Pro (M1)
• Macbook Pro (M5)

Installation

Please follow the steps in original README to setup the project in xcode.

TODOS

• Half precision inference for the MLP
• Adding different rendering mode (depth ✅, instances ✅) -> no class support right now
• Adding support to click objects
• add option to not use instance mode if clusters file is not provided (toggle disabled + warning shown)
• add optional speedup via static vs dynamic splat masking (export_static_mask.py ready, Swift integration pending)
• Update BibTex after 3DV proceedings are published
• ...

Usage

By selecting a folder in the startup page, the app loads the weights.bin, clusters.bin and point_cloud.ply inside the directory. You can download example scenes from here trained with TRASE.

There is a scroll bar for adjusting the time but you can also let it play by deactivating the manual time setting. You can toggle the additional dropdown to unlock additional options such as the TRASE based instance segmentation "Show Clusters" and depth based visualisation "Depth" (based on camera viewpoint). Clicking on clusters will isolate them in the visualisation, showing the whole scene again can be done via "Show all" button.

The gestures for X/Y Panning, Orbit, Zoom in/out are also implemented. As some scenes (such as sear-stake) may have a flipped coordinate system we add some buttons to change the coordinate axis in the dropdown.

Using your own Scenes

Step 1: Create env

conda create -n trase_model_converter python=3.10 -y
conda activate trase_model_converter
pip install torch "numpy<2" torchinfo

Step 2: For dynamic splat scenes

Output should be the base path of your input folder, where point_cloud.ply is stored.

Export the deform.pth via :

python export_deform_weights.py --model <path-to-deform.pth> --output <path to deform output.bin>

Step 3: Optionally for clusters (store in same path as point_cloud.ply just like for 1) :

Train your scene with TRASE and run the export script :

python export_clusters_bin.py --model <clusters.pt> --output <path to clusters output.bin>

Demo

Dynamic Splats + Instances on iPhone 17 Air

Iphone_sear_stake.mp4

Dynamic Splats + Instances on iPad Pro (M1)

ipad_split_cookie.mp4

Dynamic Splats + Instances on iPhone 15

as_novel_view_iPhone15.mp4

Acknowledgments

This project is a fork of MetalSplatter created by Sean Cier.

Original code is licensed under the MIT License (Copyright © 2023 Sean Cier).

Modifications and new features are licensed under MIT License (Copyright © 2026 Jim Li).

References

If you find this useful for your own work, particularly the 3D segmentation please consider taking a look at TRASE's code, paper, and website. Give us a star if you find it interesting :)

@article{li2024trase,
    title={TRASE: Tracking-free 4D Segmentation and Editing},
    author={Li, Yun-Jin and Gladkova, Mariia and Xia, Yan and Cremers, Daniel},
    journal={arXiv preprint arXiv:2411.19290},
    year={2024}
}

Contributors