[BUG FIX] Fix camera lookat ignored when entity_idx is set by Lidang-Jiang · Pull Request #2614 · Genesis-Embodied-AI/Genesis

Lidang-Jiang · 2026-03-28T14:37:36Z

Summary

Fix BaseCameraSensor.move_to_attach() silently ignoring the lookat and up parameters when a camera is attached to an entity via entity_idx.
Replace trans_to_T(pos) (translation-only) with pos_lookat_up_to_T(pos, lookat, up) to preserve the full camera orientation.
Add 3 regression tests covering attached camera lookat, equivalence with explicit offset_T, and non-regression for detached cameras.

Root cause

In BaseCameraSensor.move_to_attach(), when no explicit offset_T is provided, the offset transform was computed using trans_to_T(pos), which only encodes the position as a translation matrix and discards the rotation derived from lookat/up. This caused all entity-attached cameras to default to identity rotation regardless of their lookat target.

Fix: One-line change from trans_to_T(pos) to pos_lookat_up_to_T(pos, lookat, up), which correctly constructs a 4x4 transform encoding both position and orientation.

-            offset_T = trans_to_T(pos)
+            lookat = torch.tensor(self._options.lookat, dtype=gs.tc_float, device=gs.device)
+            up = torch.tensor(self._options.up, dtype=gs.tc_float, device=gs.device)
+            offset_T = pos_lookat_up_to_T(pos, lookat, up)

Snapshot update required

This fix changes the rendering output for test_rasterizer_attached_batched because the attached camera now correctly applies the lookat orientation instead of using identity rotation. The snapshot for this test needs to be regenerated by a maintainer with the reference hardware/GPU setup.

Before fix (lookat ignored)

[Genesis] [22:27:12] [INFO] ╭───────────────────────────────────────────────╮
[Genesis] [22:27:12] [INFO] │┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈ Genesis ┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈│
[Genesis] [22:27:12] [INFO] ╰───────────────────────────────────────────────╯
[Genesis] [22:27:13] [INFO] Running on [11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz] with backend gs.cpu. Device memory: 11.68 GB.
[Genesis] [22:27:13] [INFO] 🚀 Genesis initialized. 🔖 version: 0.4.3, 🎨 theme: dark, 🌱 seed: None, 🐛 debug: False, 📏 precision: 32, 🔥 performance: False, 💬 verbose: INFO
[Genesis] [22:27:14] [INFO] Scene <553ed15> created.
[Genesis] [22:27:14] [INFO] Adding <gs.engine.entities.RigidEntity>. idx: 0, uid: <d11c76c>, morph: <gs.morphs.Plane>, material: <gs.materials.Rigid>.
[Genesis] [22:27:14] [INFO] Adding <gs.engine.entities.RigidEntity>. idx: 1, uid: <a997cb1>, morph: <gs.morphs.Sphere>, material: <gs.materials.Rigid>.
[Genesis] [22:27:14] [INFO] Building scene <553ed15>...
[Genesis] [22:27:16] [INFO] Compiling simulation kernels...
[Genesis] [22:27:17] [INFO] Building visualizer...
Camera A image shape: (64, 64, 3), mean pixel: 102.3313
Camera B image shape: (64, 64, 3), mean pixel: 102.3313
Images identical: True
BUG CONFIRMED: lookat parameter is ignored when camera is attached to entity.
Both cameras produce the same image despite different lookat targets.
[Genesis] [22:27:26] [INFO] 💤 Exiting Genesis and caching compiled kernels...

Two cameras attached to the same entity with different lookat targets produce identical images (mean pixel: 102.3313 for both). The lookat parameter is silently ignored.

After fix (lookat correctly applied)

[Genesis] [22:27:48] [INFO] ╭───────────────────────────────────────────────╮
[Genesis] [22:27:48] [INFO] │┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈ Genesis ┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈│
[Genesis] [22:27:48] [INFO] ╰───────────────────────────────────────────────╯
[Genesis] [22:27:48] [INFO] Running on [11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz] with backend gs.cpu. Device memory: 11.68 GB.
[Genesis] [22:27:48] [INFO] 🚀 Genesis initialized. 🔖 version: 0.4.3, 🎨 theme: dark, 🌱 seed: None, 🐛 debug: False, 📏 precision: 32, 🔥 performance: False, 💬 verbose: INFO
[Genesis] [22:27:50] [INFO] Scene <c15c415> created.
[Genesis] [22:27:50] [INFO] Adding <gs.engine.entities.RigidEntity>. idx: 0, uid: <51c662e>, morph: <gs.morphs.Plane>, material: <gs.materials.Rigid>.
[Genesis] [22:27:50] [INFO] Adding <gs.engine.entities.RigidEntity>. idx: 1, uid: <d6d9e15>, morph: <gs.morphs.Sphere>, material: <gs.materials.Rigid>.
[Genesis] [22:27:50] [INFO] Building scene <c15c415>...
[Genesis] [22:27:51] [INFO] Compiling simulation kernels...
[Genesis] [22:27:53] [INFO] Building visualizer...
Camera A image shape: (64, 64, 3), mean pixel: 20.3333
Camera B image shape: (64, 64, 3), mean pixel: 50.6091
Images identical: False
FIXED: lookat parameter is correctly applied when camera is attached to entity.
The two cameras produce different images as expected.
[Genesis] [22:27:55] [INFO] 💤 Exiting Genesis and caching compiled kernels...

Two cameras with different lookat targets now produce different images (mean pixel: 20.3333 vs 50.6091), confirming the lookat parameter is correctly applied.

Unit test results (6 passed, 7 skipped)

============================= test session starts ==============================
platform linux -- Python 3.11.14, pytest-9.0.2, pluggy-1.6.0 -- /home/devuser/workspace/public-oss/embodied-robotics/myenv-genesis-2591/bin/python
cachedir: .pytest_cache
rootdir: /home/devuser/workspace/public-oss/embodied-robotics/worktree-genesis-2591
configfile: pyproject.toml
plugins: xdist-3.8.0, anyio-4.13.0, syrupy-5.1.0, forked-1.6.0
created: 1/1 worker
1 worker [13 items]

scheduling tests via WorkStealingScheduling

tests/test_sensor_camera.py::test_batch_renderer[0-cuda]
[gw0] [  7%] SKIPPED tests/test_sensor_camera.py::test_batch_renderer[0-cuda]
tests/test_sensor_camera.py::test_batch_renderer[2-cuda]
[gw0] [ 15%] SKIPPED tests/test_sensor_camera.py::test_batch_renderer[2-cuda]
tests/test_sensor_camera.py::test_destroy_unbuilt_scene_with_camera
[gw0] [ 23%] PASSED tests/test_sensor_camera.py::test_destroy_unbuilt_scene_with_camera
tests/test_sensor_camera.py::test_destroy_idempotent_with_camera
[gw0] [ 30%] PASSED tests/test_sensor_camera.py::test_destroy_idempotent_with_camera
tests/test_sensor_camera.py::test_rasterizer_destroy
[gw0] [ 38%] PASSED tests/test_sensor_camera.py::test_rasterizer_destroy
tests/test_sensor_camera.py::test_batch_renderer_destroy[cuda]
[gw0] [ 46%] SKIPPED tests/test_sensor_camera.py::test_batch_renderer_destroy[cuda]
tests/test_sensor_camera.py::test_raytracer_destroy
[gw0] [ 53%] SKIPPED tests/test_sensor_camera.py::test_raytracer_destroy
tests/test_sensor_camera.py::test_raytracer_attached_without_offset_T
[gw0] [ 61%] SKIPPED tests/test_sensor_camera.py::test_raytracer_attached_without_offset_T
tests/test_sensor_camera.py::test_raytracer[0]
[gw0] [ 69%] SKIPPED tests/test_sensor_camera.py::test_raytracer[0]
tests/test_sensor_camera.py::test_raytracer[1]
[gw0] [ 76%] SKIPPED tests/test_sensor_camera.py::test_raytracer[1]
tests/test_sensor_camera.py::test_attached_camera_lookat_affects_orientation
[gw0] [ 84%] PASSED tests/test_sensor_camera.py::test_attached_camera_lookat_affects_orientation
tests/test_sensor_camera.py::test_attached_camera_lookat_matches_offset_T
[gw0] [ 92%] PASSED tests/test_sensor_camera.py::test_attached_camera_lookat_matches_offset_T
tests/test_sensor_camera.py::test_detached_camera_lookat_still_works
[gw0] [100%] PASSED tests/test_sensor_camera.py::test_detached_camera_lookat_still_works

============================== slowest durations ===============================

(32 durations < 100s hidden.)
=========================== short test summary info ============================
SKIPPED [3] tests/conftest.py:99: Backend 'gs.cuda' not available on this machine
SKIPPED [4] tests/conftest.py:101: RayTracer is not supported because 'LuisaRenderPy' is not available.
======================== 6 passed, 7 skipped in 36.57s =========================

3 new regression tests all PASSED:
- test_attached_camera_lookat_affects_orientation -- different lookat targets produce different images
- test_attached_camera_lookat_matches_offset_T -- lookat-derived transform matches equivalent explicit offset_T
- test_detached_camera_lookat_still_works -- non-regression: detached cameras still work correctly
7 tests skipped (no CUDA/LuisaRender available on test machine)

Test plan

Reproduce bug: two attached cameras with different lookat produce identical images (before fix)
Verify fix: same cameras now produce different images (after fix)
test_attached_camera_lookat_affects_orientation -- PASSED
test_attached_camera_lookat_matches_offset_T -- PASSED
test_detached_camera_lookat_still_works -- PASSED (non-regression)
All existing non-snapshot camera tests pass
test_rasterizer_attached_batched snapshot needs regeneration by maintainer (expected due to corrected rendering)

When attaching a camera to an entity via entity_idx, the lookat and up parameters were silently ignored because move_to_attach() used trans_to_T(pos) which only sets translation, discarding orientation. Replace trans_to_T(pos) with pos_lookat_up_to_T(pos, lookat, up) to preserve the camera orientation defined by lookat/up parameters. Fixes Genesis-Embodied-AI#2591 Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>

tests/test_sensor_camera.py

- Merge 3 tests into single test_camera_lookat_entity - Remove comments, docstrings, and unused pos_lookat_up_to_T import - Add png_snapshot check for attached camera rendering Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>

tests/test_sensor_camera.py

- Remove _blurred_kernel_size (not proven necessary) - Remove surface parameter from sphere entity - Use common_options dict + for loop for camera creation - Use for loop for pair-wise image assertions Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>

duburcqa · 2026-03-29T07:02:01Z

Please share the PR on HuggingFace here.

Lidang-Jiang requested review from YilingQiao and duburcqa as code owners March 28, 2026 14:37