feat: MuJoCo simulation backend — AgentTool with 35 actions

.github/workflows/test-lint.yml

@@ -26,6 +26,11 @@ jobs:
           python-version: '3.12'
           cache: 'pip'
 
+      - name: Install system dependencies (OpenGL for MuJoCo)
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libosmesa6-dev ffmpeg
+
       - name: Install dependencies
         run: |
           pip install --no-cache-dir hatch
@@ -35,4 +40,6 @@ jobs:
         run: hatch run lint
 
       - name: Run tests
+        env:
+          MUJOCO_GL: osmesa
         run: hatch run test -x --strict-markers

.gitignore

@@ -10,3 +10,7 @@ dist
 .strands_robots
 .coverage
 .ideation/
+MUJOCO_LOG.TXT
+TASKS.md
+TASKS_TO_FIX_85.md
+.coverage.*

CHANGELOG.md

@@ -0,0 +1,187 @@
+# CHANGELOG
+
+All notable behavioural changes to `strands-robots` are logged here. Follows
+[Keep a Changelog](https://keepachangelog.com/) conventions.
+
+## Unreleased — PR #85 (MuJoCo backend remediation)
+
+### Breaking
+
+These changes tighten the MuJoCo AgentTool contract. Legacy callers that
+silently worked by accident will now receive a clear error instead:
+
+- **Router input validation**: The ``_dispatch_action`` router rejects any
+  top-level parameter that isn't declared on the target method. Passing
+  ``step(num_steps=5)`` (wrong name) or ``set_gravity(device="mps")``
+  (stray kwarg) now errors with *"Unknown parameter X for action Y.
+  Valid: [...]"* instead of silently dropping the value. Methods whose
+  Python signature includes ``**kwargs`` (e.g. ``add_object``) keep their
+  pass-through semantics.
+- **Missing required args**: produce *"Action X requires parameter Y."*
+  instead of a raw Python ``TypeError``.
+- **Vector dimension validation**: ``position``, ``target``, ``origin``,
+  ``force``, ``torque``, ``gravity``, ``direction``, ``point``, ``orientation``
+  (quaternion), and ``color`` (rgba) all validated for length + numeric
+  dtype before reaching numpy/MuJoCo.
+- **Camera orientation**: ``add_camera(target=[x,y,z])`` is now honoured
+  by baking ``xyaxes`` into the MJCF ``<camera>``. Previously the target
+  was silently dropped and every custom camera rendered a default view.
+  Degenerate case (``target == position``) errors.
+- **Render camera validation**: ``render(camera_name="missing")`` errors
+  with *"Camera 'missing' not found."* instead of silently falling back
+  to the free camera while claiming to render from the named one.
+- **Raycast zero-direction guard**: ``raycast(direction=[0,0,0])`` now
+  errors with *"direction vector is zero-length"*. Previously MuJoCo's
+  C-level ``mj_ray`` would abort the Python process.
+- **apply_force requires a non-zero vector**: passing neither ``force``
+  nor ``torque`` (or both zero) errors. Previously the call silently
+  succeeded with no effect.
+- **step(n_steps<0)** rejected (previously it corrupted ``step_count``).
+- **Negative mass / timestep / size** rejected per shape; previously
+  ``set_body_properties(mass=-1)`` and ``set_timestep(-0.01)`` silently
+  succeeded.
+- **Plane objects auto-static**: ``add_object(shape="plane")`` now forces
+  ``is_static=True`` (planes are infinite in MuJoCo). Explicit
+  ``is_static=False`` on a plane is a hard error.
+- **Duplicate camera name** rejected. Previously a second ``add_camera``
+  with an existing name silently overwrote the registry entry while
+  leaving the old camera in the XML — ghost behaviour. Use
+  ``remove_camera`` + ``add_camera`` to replace.
+- **stop_policy(robot_name='')** errors with *"stop_policy requires
+  'robot_name'."* instead of silently matching the first robot.
+- **eval_policy** requires an explicit ``robot_name``. Default
+  ``n_episodes`` lowered from 10 to 1.
+- **register_urdf** validates the path: file must exist, be a file, and
+  be readable. Previously bad paths were cached and blew up later.
+
+### Recording backend split
+
+- ``start_recording`` (LeRobotDataset: parquet + per-camera MP4) still
+  requires the ``[lerobot]`` extra. Its error message when lerobot is
+  missing now points callers at ``start_cameras_recording`` for plain
+  MP4 (which runs under ``[sim-mujoco]`` alone via imageio-ffmpeg).
+- No API change — the fix is informational.
+
+### Resource hygiene
+
+- ``destroy()`` and ``cleanup()`` now close renderers on the main thread
+  and empty the TLS cache. Previously each ``create_world/destroy``
+  cycle leaked one ``mujoco.Renderer`` + its GL context (~33 MB per
+  cycle measured). Worker-thread renderers still release themselves on
+  thread teardown (we avoid cross-thread ``close()`` to prevent
+  ``cgl.free()`` SIGSEGVs on macOS).
+- ``get_mass_matrix`` and ``get_contacts`` run ``mj_forward`` first so
+  values are valid immediately after a ``reset`` or ``add_robot``
+  (previously returned stale / uninitialised memory).
+
+### Concurrency guards
+
+Write-mutations are now refused while a policy is running on any robot
+in the world. Previously these could race the policy worker thread and
+produce undefined behaviour or SIGSEGV:
+
+    reset, set_gravity, set_timestep, set_joint_positions,
+    set_joint_velocities, apply_force, set_body_properties,
+    set_geom_properties, load_state, randomize, move_object
+
+The error now lists *which* robot(s) are active so the LLM can
+``stop_policy`` on each without guessing: *"Cannot 'X' while a policy
+is running on 'armA', 'armB'. Stop it first: action='stop_policy'."*
+
+### Concurrent per-robot policies (GH #114)
+
+Multiple ``start_policy`` calls on *different* robots now run
+concurrently. MuJoCo physics is still serialized via ``self._lock``
+(``mj_step`` and ``ctrl[]`` writes are not thread-safe for concurrent
+mutation), but each policy owns a disjoint slice of ``data.ctrl[]`` so
+two VLA arms can operate in the same scene without semantic conflict.
+
+- ``start_policy("armA")`` + ``start_policy("armB")`` both succeed.
+  Second call no longer hits a global "policy already running" gate.
+- ``start_policy`` on the *same* robot while its policy is active
+  still errors (unchanged).
+- ``remove_robot("X")`` now gracefully stops X's own policy before
+  removing, instead of requiring a prior ``stop_policy("X")``. Still
+  errors if a *different* robot has an active policy (XML round-trip
+  invalidates cached IDs everywhere).
+- New action ``list_policies_running`` returns the names of robots
+  with live policies. Prunes completed Futures as a side-effect.
+- Completed policy Futures are no longer retained forever in
+  ``_policy_threads`` (GH #120 companion fix).
+
+### Error message consistency
+
+- All "no world" paths return the same string:
+  *"No world. Call create_world (or load_scene) first."*
+- Unknown-name errors use a uniform ``<Kind> 'X' not found.`` shape
+  (Robot / Object / Body / Geom / Joint / Sensor / Camera / Checkpoint).
+- ``stop_recording``, ``stop_cameras_recording``, ``stop_policy``,
+  ``close_viewer`` are now **idempotent**: calling them when nothing
+  is running returns ``status="success"`` with a *"Was not ..."* message
+  so callers can invoke them unconditionally.
+- ``get_recording_status`` returns success in every lifecycle state
+  (no world / not recording / recording).
+
+### Deprecations
+
+- **add_robot name-as-registry fallback**: passing ``name="my_bot"``
+  without ``urdf_path`` or ``data_config`` used to resolve ``my_bot`` in
+  the model registry. This now fires a ``DeprecationWarning``. Use
+  ``add_robot(name="...", data_config="<registry_key>")`` instead. Will
+  be removed next major release.
+
+### New / extended actions
+
+- ``forward_kinematics(body_name="X")`` filters to a single body.
+- ``get_features(robot_name="X")`` filters to a single robot's joints
+  and actuators.
+- ``set_geom_properties(geom_name="X")`` accepts the bare object name
+  as an alias for the injected ``"{name}_geom"``.
+- ``render_all`` flags cameras whose frame has near-zero pixel variance
+  (``"⚠️ camera 'X': image appears empty (variance < 1)"``).
+- ``render_depth`` surfaces MuJoCo's one-time ``ARB_clip_control``
+  warning in the response text on macOS, so the LLM knows when depth
+  accuracy is reduced.
+- ``render`` / ``render_depth``: width/height validated up front;
+  oversized requests get a plain-English message naming the actual
+  framebuffer cap (``<global offwidth=...>``) instead of MuJoCo's raw
+  error.
+- ``run_policy`` / ``start_policy``: accept optional ``n_steps``
+  (primary) or legacy ``max_steps`` as an alternative to
+  ``duration``+``control_frequency``. ``duration = n_steps /
+  control_frequency`` when ``n_steps`` is set.
+- **New ``list_policies_running``** action returns the names of robots
+  with a live policy — pairs with the new concurrent-policy support
+  (see *Concurrent per-robot policies* above).
+- ``randomize(randomize_physics=True)`` now reports per-body mass scales
+  and per-geom friction scales in the response (not just range
+  endpoints).
+- ``get_contacts`` resolves unnamed geoms to
+  ``"<body_name>/geom_<id>"`` so contact pairs are always human-readable.
+- ``get_sensor_data(sensor_name="X")`` on a model with no sensors now
+  distinguishes *"Sensor 'X' not found. Model has no sensors."* from
+  the generic "no sensors in model" success.
+
+### Tests
+
+- New: ``tests/simulation/mujoco/test_agenttool_contract.py`` — ~50
+  tests that lock in router validation, tool_spec ↔ method parity,
+  unified error messages, idempotent stop family, ``mj_forward`` before
+  reads, render-dim validation, feature filters, camera duplicate
+  policy, plane auto-static, policy horizon unification, and more.
+- New: ``tests/simulation/mujoco/test_renderer_hygiene.py`` — 4 tests
+  asserting TLS cache is emptied on ``destroy``, renderer reuse works
+  for identical ``(w,h)``, and ``create_world`` after ``destroy``
+  rebuilds cleanly.
+- New: ``tests/simulation/mujoco/test_recording_backends.py`` — 2 tests
+  (one skipped when ``lerobot`` IS installed) pinning the
+  MP4-without-lerobot backend.
+- New: ``tests/simulation/mujoco/test_input_validation.py`` — 11 tests
+  for step/raycast/apply_force validation.
+- New: ``tests_integ/test_resource_hygiene.py`` — 3 integration tests
+  (require ``psutil``): 50 create/destroy cycles grow RSS < 50 MB; 500
+  renders at fixed dims grow RSS < 100 MB; TLS cache cleared on destroy.
+
+Test count: **256 → 362** (+106 new regression tests), zero
+regressions. ``hatch run lint`` (ruff + mypy) clean across 102 source
+files.

README.md

@@ -511,6 +511,108 @@ To clear the cache: `rm -rf ~/.strands_robots/assets/`
 
 To change the cache location: `export STRANDS_ASSETS_DIR=/path/to/custom/dir`
 
+## Simulation (MuJoCo)
+
+`strands-robots` ships a MuJoCo-backed simulation AgentTool — 58 actions
+exposed to any Strands agent for world composition, physics, policy
+execution, and video/dataset recording.
+
+### Install
+
+```bash
+pip install "strands-robots[sim-mujoco]"
+# For LeRobotDataset recording (parquet + training data):
+pip install "strands-robots[sim-mujoco,lerobot]"
+```
+
+### Quick start
+
+```python
+from strands_robots.simulation import Simulation
+
+sim = Simulation(tool_name="sim", mesh=False)
+sim.create_world()
+sim.add_robot(name="arm", data_config="so100")
+sim.add_object(name="cube", shape="box", position=[0.3, 0, 0.05])
+sim.add_camera(name="topdown", position=[0, 0, 1.5], target=[0, 0, 0])
+
+sim.run_policy(robot_name="arm", policy_provider="mock", n_steps=200,
+               control_frequency=50.0, fast_mode=True)
+
+frame = sim.render(camera_name="topdown")  # returns {status, content:[text, image]}
+```
+
+### 58 actions grouped
+
+- **World & objects**: `create_world`, `load_scene`, `add_robot`,
+  `add_object`, `move_object`, `list_objects`, `list_robots`,
+  `remove_robot`, `remove_object`, `destroy`, `reset`, `get_state`,
+  `save_state`, `load_state`, `list_checkpoints`.
+- **Physics**: `step`, `set_timestep`, `set_gravity`, `apply_force`,
+  `raycast`, `multi_raycast`, `set_body_properties`,
+  `set_geom_properties`, `get_body_state`, `get_joint_state`,
+  `set_joint_positions`, `set_joint_velocities`, `forward_kinematics`,
+  `get_mass_matrix`, `inverse_dynamics`, `get_total_mass`,
+  `get_jacobian`, `get_energy`, `get_contacts`, `get_sensor_data`.
+- **Cameras & rendering**: `add_camera`, `remove_camera`, `render`,
+  `render_depth`, `render_all`, `start_cameras_recording`,
+  `stop_cameras_recording`, `get_cameras_recording_status`.
+- **Policy**: `start_policy`, `run_policy`, `stop_policy`,
+  `replay_episode`, `eval_policy`.
+- **Randomization**: `randomize`.
+- **Recording (LeRobotDataset)**: `start_recording`, `stop_recording`,
+  `get_recording_status`.
+- **Introspection & util**: `get_features`, `list_urdfs`, `register_urdf`,
+  `export_xml`, `open_viewer`, `close_viewer`.
+
+### Common footguns
+
+- **Planes must be static.** `add_object(shape="plane")` auto-sets
+  `is_static=True`. Passing `is_static=False` on a plane is a hard error
+  (MuJoCo planes are infinite and can't have dynamic mass).
+- **Camera orientation.** Pass `target=[x,y,z]` to look at a point —
+  without it the camera faces forward by default. `target == position`
+  errors.
+- **MP4 vs dataset recording.** `start_cameras_recording` writes plain
+  MP4 per-camera and runs under `[sim-mujoco]` alone. `start_recording`
+  writes a LeRobotDataset (parquet + MP4 + schema) and requires the
+  `[lerobot]` extra.
+- **Policy running → mutations blocked.** While a policy runs on any
+  robot, state-mutating actions (`reset`, `set_gravity`, joint setters,
+  `apply_force`, `set_body_properties`, `set_geom_properties`,
+  `load_state`, `randomize`, `move_object`) error with *"Cannot 'X'
+  while a policy is running."* Stop it first with
+  `stop_policy(robot_name='...')`.
+- **Horizon parameters.** `run_policy` accepts either `duration` +
+  `control_frequency` (real-time) OR `n_steps` + `control_frequency`
+  (step-count). Pass `fast_mode=True` to skip the between-step sleep
+  during batch eval / data collection.
+- **Name collisions.** Objects, bodies, robots, and cameras share the
+  MuJoCo name table. Robot joints and actuators are auto-namespaced as
+  `{robot_name}/{joint}` in multi-robot scenes. Object geoms are
+  injected as `{object_name}_geom`; `set_geom_properties` accepts the
+  bare object name as an alias.
+- **Oversized render**: MuJoCo's offscreen framebuffer is capped by
+  `<global offwidth="W" offheight="H"/>` in MJCF. Requesting a bigger
+  render now errors with a plain message naming the cap — either lower
+  the request or rebuild the model with larger dims.
+
+### Self-healing features
+
+- Unknown parameters are rejected with *"Unknown parameter X for action
+  Y. Valid: [...]"* so the LLM learns the correct name without trial-
+  and-error.
+- Missing required parameters produce *"Action X requires parameter Y."*
+  (no Python `TypeError` leaks).
+- Vector dimensions and numeric dtype are validated before MuJoCo sees
+  them (previously zero-length direction vectors crashed the Python
+  process via `mj_ray` C-level abort).
+- `destroy()` and `cleanup()` empty the renderer TLS cache and shut down
+  the executor — no RSS growth across repeated create/destroy cycles.
+
+For the full action contract and test coverage see
+`tests/simulation/mujoco/test_agenttool_contract.py`.
+
 ## Contributing
 
 We welcome contributions! Please see:

pyproject.toml

@@ -48,13 +48,16 @@ groot-service = [
 lerobot = [
     "lerobot>=0.5.0,<0.6.0",
 ]
-sim = [
+sim-mujoco = [
     "robot_descriptions>=1.11.0,<2.0.0",
+    "mujoco>=3.0.0,<4.0.0",
+    "imageio>=2.28.0,<3.0.0",
+    "imageio-ffmpeg>=0.4.0,<1.0.0",
 ]
 all = [
     "strands-robots[groot-service]",
     "strands-robots[lerobot]",
-    "strands-robots[sim]",
+    "strands-robots[sim-mujoco]",
 ]
 dev = [
     "pytest>=6.0,<9.0.0",
@@ -79,6 +82,7 @@ packages = ["strands_robots"]
 
 [tool.hatch.envs.default]
 installer = "uv"
+features = ["all"]
 dependencies = [
     "pytest>=6.0,<9.0.0",
     "pytest-cov>=4.0.0,<6.0.0",
@@ -128,7 +132,7 @@ ignore_missing_imports = false
 
 # Third-party libs without type stubs
 [[tool.mypy.overrides]]
-module = ["lerobot.*", "gr00t.*", "draccus.*", "msgpack.*", "zmq.*", "huggingface_hub.*", "serial.*", "psutil.*", "torch.*", "torchvision.*", "transformers.*", "einops.*", "robot_descriptions.*"]
+module = ["lerobot.*", "gr00t.*", "draccus.*", "msgpack.*", "zmq.*", "huggingface_hub.*", "serial.*", "psutil.*", "torch.*", "torchvision.*", "transformers.*", "einops.*", "robot_descriptions.*", "mujoco.*", "imageio.*"]
 ignore_missing_imports = true
 
 # @tool decorator injects runtime signatures mypy cannot check
@@ -161,6 +165,22 @@ module = ["strands_robots.registry.*"]
 warn_return_any = false
 disallow_untyped_defs = false
 
+# MuJoCo simulation — mixins use cooperative self._world patterns
+# attr-defined: Mixins access self._world/self._lock/etc. from Simulation (cooperative pattern)
+# assignment: PEP 484 implicit Optional (= None on typed params)
+# override: Subclass signatures extend base with extra params (orientation, mesh_path)
+# misc: Multiple inheritance method resolution conflicts between mixin + ABC
+[[tool.mypy.overrides]]
+module = ["strands_robots.simulation.mujoco.*"]
+disallow_untyped_defs = false
+warn_return_any = false
+
+# Async utils and dataset recorder — thin wrappers with dynamic types
+[[tool.mypy.overrides]]
+module = ["strands_robots._async_utils", "strands_robots.dataset_recorder"]
+disallow_untyped_defs = false
+warn_return_any = false
+
 # Test files — relaxed type checking for mocks, fixtures, and test utilities
 [[tool.mypy.overrides]]
 module = ["tests.*", "tests_integ.*"]

strands_robots/__init__.py

@@ -25,14 +25,11 @@
 import warnings as _warnings
 from typing import Any
 
-# ------------------------------------------------------------------
 # Light-weight imports — no torch / lerobot dependency
-# ------------------------------------------------------------------
 from strands_robots.policies import MockPolicy, Policy, create_policy  # noqa: F401
 
-# ------------------------------------------------------------------
 # Lazy-loaded heavy symbols
-# ------------------------------------------------------------------
+
 # Maps public name -> (module_path, attribute_name)
 _LAZY_IMPORTS: dict[str, tuple[str, str]] = {
     "Robot": ("strands_robots.robot", "Robot"),