feat(isaac): Isaac Sim simulation backend — USD + IsaacLab 3.0

[cagataycali]

Summary

Implement IsaacSimulation(SimEngine) — a photorealistic, USD-native simulation backend built on NVIDIA Isaac Sim + IsaacLab 3.0. Third concrete SimEngine subclass (after MuJoCo #85 and Newton #TBD), targeted at photorealistic training data generation, teleoperation with real-hardware-fidelity scenes, and SIL/HIL pipelines (#5).

Isaac Sim's differentiating value is environment fidelity — USD scenes, Omniverse materials, PhysX GPU, and ray-traced rendering. This is the backend that powers synthetic-dataset generation pipelines (DreamGen / Replicator).

---

Motivation

Capability	MuJoCo (#85)	Newton (#TBD)	Isaac Sim (this issue)
Photorealistic rendering	❌	basic OpenGL	✅ RTX path-traced
USD scene format	❌	partial	✅ native
Omniverse material library	❌	❌	✅
Synthetic data generation	❌	❌	✅ (Replicator)
IsaacLab RL environments	❌	❌	✅
Setup complexity	low	medium	high
macOS support	✅	❌	❌
Download size	~50 MB	~500 MB	~30 GB

Isaac Sim fills the "photorealistic synthesis" slot. It is strictly opt-in — most contributors won't install it. We guard everything behind [isaac] extras and a runtime is_available() check.

Suggested release target: v0.4.0 if it fits; acceptable to slip to v0.4.1. The asset converter sub-PR can merge independently without the Isaac Sim runtime.

---

Design

Directory layout

strands_robots/simulation/isaac/
├── __init__.py              # Export IsaacSimulation
├── simulation.py            # class IsaacSimulation(SimEngine)
├── config.py                # @dataclass IsaacSimConfig
├── asset_converter.py       # urdf_to_usd / mjcf_to_usd (no Isaac runtime dep)
├── bridge.py                # State sync bridge
├── lab/                     # IsaacLab-specific helpers
│   ├── __init__.py
│   ├── env.py               # IsaacLab env adapter
│   └── trainer.py           # RL training wrapper
└── tests/
    ├── test_unit.py         # No Isaac deps — runs in main CI
    └── test_integ.py        # @pytest.mark.isaac — nightly GPU runner

Core class

from strands_robots.simulation.base import SimEngine
from strands_robots.simulation.isaac.config import IsaacSimConfig

class IsaacSimulation(SimEngine):
    """Photorealistic USD-native simulation backend on NVIDIA Isaac Sim.
    
    Requires Isaac Sim 5.0+, IsaacLab 3.0+, RTX GPU, and ~30 GB disk.
    Imported lazily — importing this module does NOT load Isaac Sim.
    """

    def __init__(self, config: IsaacSimConfig | None = None) -> None: ...

    def is_available(self) -> bool:
        """Return True if Isaac Sim SDK is importable on this system."""

    # --- Required SimEngine methods ---
    def create_world(self, timestep=None, gravity=None, ground_plane=True) -> dict: ...
    def destroy(self) -> dict: ...
    def reset(self, env_ids=None) -> dict: ...
    def step(self, n_steps=1) -> dict: ...
    def get_state(self) -> dict: ...
    def add_robot(self, name, urdf_path=None, data_config=None,
                  position=None, orientation=None) -> dict: ...
    def remove_robot(self, name) -> dict: ...
    def add_object(self, name, shape="box", **kwargs) -> dict: ...
    def remove_object(self, name) -> dict: ...
    def get_observation(self, robot_name=None, camera_name=None) -> dict: ...
    def send_action(self, action, robot_name=None, n_substeps=1) -> None: ...
    def render(self, camera_name="default", width=None, height=None) -> dict: ...

    # --- Optional overrides ---
    def load_scene(self, scene_path: str) -> dict: ...  # USD
    def run_policy(self, robot_name, policy_provider="gr00t", **kwargs) -> dict: ...
    def get_contacts(self) -> dict: ...

    # --- Isaac-specific extensions ---
    def add_terrain(self, terrain_type="flat_plane", **kwargs) -> dict: ...
    def set_joint_positions(self, robot_name, positions: dict) -> dict: ...
    def get_contact_forces(self, robot_name) -> dict: ...
    def set_camera_pose(self, camera_name, position, target) -> dict: ...
    def record_video(self, output_path, duration=10.0, fps=30) -> dict: ...
    def _resolve_usd(self, data_config: str) -> Path | None:
        """Resolve robot name to on-disk USD path."""

IsaacSimConfig

@dataclass
class IsaacSimConfig:
    headless: bool = True                # Set False for GUI dev
    device: str = "cuda:0"
    physics_dt: float = 1.0 / 60.0
    rendering_dt: float = 1.0 / 30.0
    enable_livestream: bool = False      # Omniverse streaming
    isaac_sim_path: str | None = None    # Override auto-detect
    replicator_enabled: bool = False     # For synthetic data
    rtx_mode: str = "path_traced"        # path_traced | real_time | pbr

---

Definition of Done

Functional

• create_simulation("isaac") returns IsaacSimulation when Isaac Sim is installed
• create_simulation("isaac") raises a helpful error with install URL when missing
• sim.is_available() returns bool safely at runtime
• sim.add_robot("so100", data_config="so100") finds and loads USD
• sim.add_robot("unitree_g1") works with packaged USD
• sim.step(100) stable
• sim.get_observation("so100") returns joint + camera dict
• sim.render() returns RGB frame (path-traced or real-time per config)
• Teleoperation loop works: input → set_joint_positions → step

Advanced

• sim.load_scene("path/to/scene.usd") works with IsaacLab environments
• sim.add_terrain("rough", **params) generates procedural terrain
• sim.get_contact_forces("g1") returns per-link force vectors
• sim.record_video(output="demo.mp4", duration=10) works end-to-end
• Replicator synthetic-data path wired up

Asset converter

• urdf_to_usd(urdf_path, out_path) works for so100, g1, go2
• mjcf_to_usd(mjcf_path, out_path) works for MuJoCo Menagerie assets
• Auto-converts robots that don't ship with USD
• Conversion is idempotent and cached at ~/.cache/strands-robots/usd/
• Sub-PR mergeable without Isaac Sim runtime dependency

Registry + API

• factory._BUILTIN_BACKENDS["isaac"] = ("strands_robots.simulation.isaac.simulation", "IsaacSimulation")
• Aliases: "isaac_sim", "isaacsim", "nvidia"
• Robot("so100", mode="sim", backend="isaac") works

Quality

• pip install strands-robots[isaac] resolves (pinned Isaac Sim deps via extras)
• IsaacLab 3.0 breaking changes from Track IsaacLab 3.0 breaking changes — quaternion, wp.array, multi-backend #68 applied (quat, wp.array, multi-backend)
• Lazy imports — import strands_robots must NOT load Isaac Sim
• Unit tests pass WITHOUT Isaac Sim installed (mocked)
• @pytest.mark.isaac integration suite on dedicated GPU+Isaac CI runner (nightly)
• NumPy-style docstrings + type hints on every public method

Documentation

• docs/backends/isaac.md — installation, IsaacLab setup, troubleshooting
• examples/isaac_photo_realistic_so100.py — grasp demo with RTX
• examples/isaac_teleop_g1.py — joystick-driven humanoid teleop
• examples/isaac_synthetic_dataset.py — Replicator-based data gen
• Clear system requirements in README (RTX GPU, 30 GB disk, Ubuntu 22.04+)

---

Non-goals

• Full Omniverse integration — standalone Isaac Sim only
• Live-stream to Omniverse Cloud (post-v0.5)
• Custom USD material authoring — users bring their own USD
• Apple Silicon / Windows (Linux only; Isaac Sim requirement)
• Distributed multi-machine (single machine only)

---

Implementation plan (5 PRs)

1. feat(isaac): stub IsaacSimulation(SimEngine) + registry — skeleton with is_available(), all methods raise NotImplementedError. Works without Isaac Sim installed. ~200 LOC.
2. feat(isaac): asset converter (urdf_to_usd / mjcf_to_usd) — no Isaac Sim runtime dependency. Mergeable independently. ~1,500 LOC.
3. feat(isaac): world lifecycle + robot loading — requires Isaac Sim runtime. Integration test. ~800 LOC.
4. feat(isaac): observation/action/rendering — obs, actions, camera, RTX render. ~600 LOC.
5. feat(isaac): teleop + terrain + contact forces — advanced extensions. ~500 LOC.

Total estimated: ~3.6K LOC + ~1.5K tests + ~1K docs.

---

Risks & mitigations

Risk	Severity	Mitigation
Isaac Sim SDK version churn (5.0 → 6.0)	HIGH	Pin isaacsim==5.* in extras; test matrix
IsaacLab API churn (#68 was an example)	HIGH	Abstract behind internal adapter; flag breakages fast
30 GB download scares contributors	MEDIUM	Strictly opt-in via [isaac] extras; never import by default
USD asset licensing	MEDIUM	Ship converters, not pre-converted assets; MIT/Apache-licensed examples only
CI cost (GPU runner required)	MEDIUM	@pytest.mark.isaac nightly workflow, not per-PR

---

Acceptance test

# Requires Isaac Sim 5.0+ + IsaacLab 3.0+
python -c "
from strands_robots.simulation import create_simulation

sim = create_simulation('isaac', headless=True, rtx_mode='real_time')
assert sim.is_available()
sim.create_world()
sim.add_robot('so100')
sim.add_object('cube', shape='box', position=[0.3, 0, 0.05], color=[1,0,0,1])

for _ in range(60):
    obs = sim.get_observation('so100', camera_name='wrist')
    print('camera image:', obs['image'].shape)
    sim.step(1)

sim.record_video('output.mp4', duration=5)
sim.destroy()
"

---

Related

• Depends on: feat: simulation foundation — models, ABC, factory, model registry, assets #84 (SimEngine ABC), feat: MuJoCo simulation backend — AgentTool with 35 actions #85 (MuJoCo reference backend)
• Enables: SIL/HIL Integration with Isaac Sim #5 (SIL/HIL), DreamGen pipelines
• IsaacLab 3.0 migration context: Track IsaacLab 3.0 breaking changes — quaternion, wp.array, multi-backend #68
• Roadmap: strands-labs/robots roadmap | v0.3.8 → v0.3.9 → v0.4.0 #94
• Isaac Sim upstream: https://developer.nvidia.com/isaac-sim
• IsaacLab upstream: https://github.com/isaac-sim/IsaacLab

[yinsong1986]

FYI: As we discussed, shall we Implement this as a major upgrade to https://github.com/strands-labs/robots-sim, keep it consistent with https://github.com/strands-labs/robots? @cagataycali

[cagataycali]

FYI: As we discussed, shall we Implement this as a major upgrade to https://github.com/strands-labs/robots-sim, keep it consistent with https://github.com/strands-labs/robots? @cagataycali

I believe we should implement in strands-labs/robots-sim ^^

We can shepherd the code in this repository and meanwhile prepare the codebase to ready for the PR?