From 18435a17ae2921cd537d9b5db55e919d93d641ac Mon Sep 17 00:00:00 2001 From: Robin Lindh Date: Wed, 25 Mar 2026 12:55:40 +0100 Subject: [PATCH 1/2] chore: add node documentation --- CONTRIBUTING.md | 57 +++++++++++++-- README.md | 73 +++++-------------- docs/deployment.md | 137 +++++++++++++++++++++++++++++++++++ docs/minimum-requirements.md | 17 +++++ 4 files changed, 221 insertions(+), 63 deletions(-) create mode 100644 docs/deployment.md create mode 100644 docs/minimum-requirements.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 83dbcee..e8004b7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,12 +5,7 @@ Thank you for your interest in contributing. We welcome issues and pull requests ## How to contribute - **Bug reports and feature ideas:** Open an [issue](https://github.com/aukilabs/splatter-server/issues). -- **Code changes:** Open a pull request (PR) against `main`. Use a branch name that matches our CI (e.g. `feature/your-feature`, `bug/fix-description`, `chore/your-change`). - -## Development setup - -- **Rust (compute node):** See [server/rust/README.md](server/rust/README.md). Copy `server/rust/.env.example` to `server/rust/.env` and configure for local DDS/DMS if needed. -- **Docker:** From the repo root, `docker build -t splatter-server .` then run with `--env-file .env` as in the main [README](README.md). +- **Code changes:** Open a pull request (PR) against `develop`. Use a branch name that matches our CI (e.g. `feature/your-feature`, `fix/fix-description`, `chore/your-change`). ## Code standards @@ -22,8 +17,56 @@ Thank you for your interest in contributing. We welcome issues and pull requests ## Pull request process -1. Point your PR at the `main` branch. +1. Point your PR at the `develop` branch. 2. Ensure CI passes (Rust format, clippy, tests). 3. Keep changes focused; link related issues where applicable. By contributing, you agree that your contributions will be licensed under the same [MIT License](LICENSE) that covers this project. + +# Local Development + +## Development setup + +- **Rust (compute node):** See [server/rust/README.md](server/rust/README.md). Copy `server/rust/.env.example` to `server/rust/.env` and configure for local DDS/DMS if needed. +- **Docker:** From the repo root, `docker build -t splatter-server .` then run with `--env-file .env`. + +## Run Trainer +```bash +python3 run.py \ +--domain_id {domain_id} +--job_id {job_id} \ +--job_root_path {path/to/job/root} \ +--log_level {log level} +``` + +## Input Files +```bash +# Input Files +{job_root_path} +├── datasets +│ └── {dataset} +│ └── Frames.mp4 +├── refined +│ └── global +│ └── refined_sfm_combined +│ ├── cameras.bin +│ ├── images.bin +│ └── points3D.bin +``` +## Output Files +```bash +# Output Files +{job_root_path} +├── Frames +│ ├── {images} +│ └── ... +├── refined +│ ├── nerfstudio-data +│ │ └── {converted nerfstudio data from colmap} +│ └── splatter +│ ├── splat.ply +│ ├── splat_rot.ply +│ ├── splat_rot.splat # this is what needs to be uploaded to dmt +│ └── splatfacto +│ └── {splat torch model} +``` diff --git a/README.md b/README.md index 60be944..95538f5 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,14 @@ -# splatter-server +# Splatter Node +This repository contains the splatter node, part of the Auki Network. This node enables photorealistic scene rendering by training 3D Gaussian Splats. -Splatter compute node for the Auki Network: runs Gaussian splatting jobs (COLMAP + nerfstudio/splatfacto) as part of the reconstruction pipeline. +The splatter node operates in conjunction with the [reconstruction node](https://github.com/aukilabs/reconstruction-server) and the scans from the Domain Management Tool (DMT) app ([App Store](https://apps.apple.com/app/domain-management-tool/id6499270503) 🔗). The refined camera poses from the reconstruction node are used as a starting point for training the gaussian splat, making it more robust to challenging indoor environments and noisy captures. + +For more information about the reconstruction and rendering pipeline, please refer to our [whitepaper](https://auki.gitbook.io/whitepaper/technical-overview/the-reconstruction-service). + +## Documentation +- [Minimum Requirements](docs/minimum-requirements.md) +- [Deployment](docs/deployment.md) +- [Contributing](CONTRIBUTING.md) ## License @@ -10,59 +18,12 @@ This project is licensed under the [MIT License](LICENSE). See [CONTRIBUTING.md](CONTRIBUTING.md) for how to report issues, open PRs, and run the Rust checks locally. -## Docker - -### Build -From repo root path: -```bash -docker build -t splatter-server . -``` - -### Run - -Create a `.env` from `server/rust/.env.example` and set your DMS/DDS URLs and registration secret, then: - -```bash -docker run --gpus all -p 8080:8080 --name splatter -d --env-file .env splatter-server -``` +## Acknowledgments -## Run Trainer -```bash -python3 run.py \ ---domain_id {domain_id} ---job_id {job_id} \ ---job_root_path {path/to/job/root} \ ---log_level {log level} -``` +This project builds upon the work of many excellent open-source projects, including +[nerfstudio](https://github.com/nerfstudio-project/nerfstudio), [ply2splat](https://github.com/bastikohn/ply2splat), [PyTorch](https://pytorch.org), +[OpenCV](https://opencv.org), [Open3D](https://www.open3d.org), and others. -## Required Files -```bash -# Input Files -{job_root_path} -├── datasets -│ └── {dataset} -│ └── Frames.mp4 -├── refined -│ └── global -│ └── refined_sfm_combined -│ ├── cameras.bin -│ ├── images.bin -│ └── points3D.bin -``` -## Output Files -```bash -# Output Files -{job_root_path} -├── Frames -│ ├── {images} -│ └── ... -├── refined -│ ├── nerfstudio-data -│ │ └── {converted nerfstudio data from colmap} -│ └── splatter -│ ├── splat.ply -│ ├── splat_rot.ply -│ ├── splat_rot.splat # this is what needs to be uploaded to dmt -│ └── splatfacto -│ └── {splat torch model} -``` +We thank their authors and contributors for making this work possible. +Please note that all third-party code and libraries are subject to their respective licenses, copyrights, and trademarks. +We are not affiliated with, endorsed by, or sponsored by any of the projects or organizations mentioned above. \ No newline at end of file diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..b964887 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,137 @@ +# Deployment + +The splatter node is available on [Docker Hub](https://hub.docker.com/r/aukilabs/splatter-server). Both deployment options are Docker-based and still run the same splatter logic under the hood. + +## Initial Setup + +1. First check that your NVIDIA driver and CUDA toolkit meet the requirements in the [Minimum Requirements](minimum-requirements.md) and update as needed: + ```shell + nvidia-smi + ``` + This should output information about your GPU. If not, please double check that your installed CUDA and driver versions are correct, and restart your computer. + +2. Ensure outbound HTTPS access to DDS/DMS endpoints from this host. + +3. Disable power-saving settings like automatic sleep or standby mode, to keep your computer on and able to receive jobs. + +## Prepare credentials + env file + +Before you run the container, create a `.env` file based on the example file `server/rust/.env.example` with the required credentials: +```shell +REG_SECRET=your-registration-secret +SECP256K1_PRIVHEX=your-evm-private-key +``` +Optional overrides (defaults shown): +```shell +DMS_BASE_URL=https://dms.auki.network/v1 +DDS_BASE_URL=https://dds.auki.network +REQUEST_TIMEOUT_SECS=60 +REGISTER_INTERVAL_SECS=120 +REGISTER_MAX_RETRY=-1 +LOG_FORMAT=text +``` + +Notes: +- `REG_SECRET` comes from registering a **compute node** in the Posemesh Console. +- `SECP256K1_PRIVHEX` is the hex-encoded private key of the **staked** EVM wallet for that node. + +How to get the registration secret + wallet key: + +⚠️ **IMPORTANT:** If you run both the reconstruction and splatter nodes, you need to stake them both, on different wallets. + +1. Log in to the Posemesh Console at `https://console.auki.network/`. +2. Open the **Manage Nodes** page and create a compute node (choose the appropriate operation mode). +3. Copy the registration credentials shown — you will use these as `REG_SECRET`. +4. Go to **Staking**, connect your wallet, and stake the required amount of $AUKI for that node. +5. Export the wallet private key (hex) and set it as `SECP256K1_PRIVHEX`. + +## Option 1 — Use the prebuilt image (recommended) + +Pull the latest stable docker image. If you are upgrading from a previous version you must **run this again** to pull the updated image. +```shell +docker pull aukilabs/splatter-server:stable +``` + +Start Docker using the below command, ❗**including all flags**❗ +```shell +docker run \ + --gpus all \ + --env-file .env \ + --name splatter-server \ + -d \ + aukilabs/splatter-server:stable +``` +You can also pin a specific release tag, for example `aukilabs/splatter-server:vX.Y.Z`. + +### Verification ✅ + +1. After deploying, please ensure the server started correctly by running + ```shell + docker ps + ``` + This should show your newly started docker container, with the STATUS showing `Up 45 seconds` or similar. + Copy the container ID of your server, then run: + ```shell + docker logs + ``` + You should see logs indicating the node is registering with DDS and polling DMS. + +2. Ensure your GPU and CUDA works correctly (using the container ID from above): + ```shell + docker exec nvidia-smi + ``` + This should show your GPU and driver information. + + Verify that torch detects your GPU: + ```shell + docker exec python3 -c "import torch; print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'CUDA not found')" + ``` + + If not, please double-check your setup, or see **Troubleshooting**. + +## Option 2 — Build Docker image from source + +### Building Docker + +```bash +docker build -t splatter-server . +``` + +Run the image as described in Option 1, and follow the same verification steps. + +## Troubleshooting ⚠️ + +Here are some common issues you may encounter, with suggested fixes: + +### GPU not detected +- **Symptom:** Server starts but jobs fail, GPU not utilised, `CUDA not found`, or similar. +- **Fixes:** + - Ensure you started the Docker container with `--gpus all`. + - Check your driver and CUDA toolkit versions against [Minimum Requirements](minimum-requirements.md). + - Run `nvidia-smi` on host to confirm your GPU is visible. + - Run `nvcc --version` to confirm your CUDA toolkit is installed, and with the correct version. + - Restart your computer and try again. + +### Wrong GPU used +- **Symptom:** The Docker container is using the wrong GPU, for example the integrated GPU instead of the discrete GPU. +- **Fixes:** + - Check your system activity to see which GPU is being used. + - Specify the correct GPU in the Docker run command using the `--gpus` flag + +### Container killed or crashes under load +- **Symptom:** Server stops or computer restarts during job processing +- **Fix:** + - Monitor system RAM and temperatures, and check for overheating or insufficient resources. + +### Docker crashes on Windows +- **Symptom:** Container stops after a few minutes on Windows. +- **Fix:** + - Restart Docker Desktop. + - If the issue persists, also restart your computer. + +--- + +💡 **Still stuck?** +If your issue remains, please: +1. Check `docker logs ` for error messages. +2. Share logs and system specs with the [Auki](https://www.auki.com) team for support. diff --git a/docs/minimum-requirements.md b/docs/minimum-requirements.md new file mode 100644 index 0000000..990f09f --- /dev/null +++ b/docs/minimum-requirements.md @@ -0,0 +1,17 @@ +# Minimum Requirements + +The splatter node has fairly high requirements as the training of a realistic gaussian splat is computationally intensive. Please ensure your system is able to handle high CPU and GPU load for extended periods without overheating. + +**NOTE:** The requirements below are high to ensure smooth processing. If you successfully run on lower hardware specs, please let us know! + +- **OS (64-bit):** Windows 10 / 11 or Ubuntu 22.04 / 24.04 LTS +- **CPU:** 4 cores +- **RAM:** 12 GiB (recommended 16 GiB or more) +- **GPU:** Nvidia with 8+ GiB VRAM. Tested on RTX 3090, RTX 4060 and T4. RTX 50xx is not currently supported, but planned for upcoming releases. May work on older Nvidia cards too with enough VRAM and recent CUDA. +- **NVIDIA driver:** Recent driver with CUDA 11.8 support. +- **Disk space:** 40 GB or more +- **Docker** _- Windows support tested with Docker Desktop and WSL 2_ +- Sufficient cooling and power supply to handle high load. +- A stable Internet connection with at least 10 Mbps downstream and upstream + +See [Deployment](deployment.md) for more information. \ No newline at end of file From db942931821ef31be21bc6d25cd6528b7d8e20aa Mon Sep 17 00:00:00 2001 From: Robin Lindh Date: Wed, 25 Mar 2026 15:29:45 +0100 Subject: [PATCH 2/2] chore: clarify troubleshooting docs for GPU index (thanks Ali G for suggesting this) --- docs/deployment.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/deployment.md b/docs/deployment.md index b964887..cf80fe0 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -116,7 +116,8 @@ Here are some common issues you may encounter, with suggested fixes: - **Symptom:** The Docker container is using the wrong GPU, for example the integrated GPU instead of the discrete GPU. - **Fixes:** - Check your system activity to see which GPU is being used. - - Specify the correct GPU in the Docker run command using the `--gpus` flag + - Run `nvidia-smi -L` to list all GPUs on your system. + - Run the container with the correct GPU index specified in the `--gpus` flag, for example `docker run --gpus '"device=0"' ...`. ### Container killed or crashes under load - **Symptom:** Server stops or computer restarts during job processing