SkillGen in Isaac Lab for 2.3 Release #3303

njawale42 · 2025-08-28T18:32:18Z

Description

This PR introduces the SkillGen framework to Isaac Lab, integrating GPU motion planning with skill-segmented data generation. It enables efficient, high-quality dataset creation with robust collision handling, visualization, and reproducibility.

Note:

Please look at the cuRobo usage license
Please look at updated isaacsim license

Technical Implementation:

Annotation Framework:

Manual subtask start annotations to cleanly separate skill execution from motion-planning segments
Consistent trajectory segmentation for downstream dataset consumers

Motion Planning:

Base Motion Planner (Extensible):
- Introduces a reusable planner interface for uniform integration:
  - source/isaaclab_mimic/isaaclab_mimic/motion_planners/base_motion_planner.py
- Defines a minimal, consistent API for planners:
  - update_world_and_plan_motion(...), get_planned_poses(...), etc.
- The cuRobo planner inherits from this base class.
- New planners can be added by subclassing the base class and implementing the same API, enabling drop-in replacement without changes to the SkillGen pipeline.
CuRobo Planner (GPU-accelerated, collision-aware):
- Multi-phase planning: approach → contact → retreat
- Dynamic object attach/detach and contact-aware sphere management
- Real-time world synchronization between Isaac Lab and cuRobo
- Configurable collision filtering for contact phases
- Tests:
  - source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py
  - source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py
  - source/isaaclab_mimic/test/test_generate_dataset_skillgen.py

Data Generation Pipeline:

Automated dataset generation with precise skill-based segmentation
Integrates with existing observation/action spaces
Supports multi-env parallel collection with cuRobo-backed planning

Visualization and Debugging:

Rerun-based 3D visualization for trajectory/collision inspection
Real-time sphere visualization for collision boundaries and contact phases

Dependencies:

cuRobo: motion planning and collision checking
Rerun: 3D visualization and debugging

Integration:

This extends the existing mimic pipeline and remains backward compatible. It integrates into the manager-based environment structure and existing observation/action interfaces without breaking current users.

Type of change

New feature (non-breaking change which adds functionality)
This change requires a documentation update

Screenshot

SkillGen Data Generation

Cube Stacking SkillGen Data Generation	Bin Cube Stacking SkillGen Data Generation (Using Vanilla Cube Stacking Source Demos)

Bin Cube Stacking Behavior Cloned Policy

Rerun Integration

Motion Planner Tests

Obstacle Avoidance (cuRobo)	Cube Stack End-to-End (cuRobo)

Checklist

I have run the pre-commit checks with ./isaaclab.sh --format
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
I have updated the changelog and the corresponding version in the extension's config/extension.toml file
I have added my name to the CONTRIBUTORS.md or my name already exists there

…or changes in cuRobo. Added urdf extraction and on-the-fly config creation from nucleus.

gavrielstate · 2025-08-28T21:00:46Z

README.md

@@ -188,6 +188,10 @@ where creativity and technology intersect. Your contributions can make a meaning

 The Isaac Lab framework is released under [BSD-3 License](LICENSE). The `isaaclab_mimic` extension and its corresponding standalone scripts are released under [Apache 2.0](LICENSE-mimic). The license files of its dependencies and assets are present in the [`docs/licenses`](docs/licenses) directory.

+Note that Isaac Lab requires Isaac Sim, which includes components under proprietary licensing terms. Please see the [Isaac Sim license](https://github.com/isaac-sim/IsaacSim/blob/main/LICENSE) for information on Isaac Sim licensing.


This should just point to the docs/licenses/dependencies/isaacsim-license.txt license file, which should be updated with the contents of https://github.com/isaac-sim/IsaacSim/blob/main/LICENSE

Corrected! Thanks Gav!

scripts/imitation_learning/isaaclab_mimic/annotate_demos.py

nv-caelan · 2025-08-28T21:44:36Z

scripts/imitation_learning/isaaclab_mimic/annotate_demos.py

+class PreStepSubtaskStartsObservationsRecorder(RecorderTerm):
+    """Recorder term that records the subtask start observations in each step."""
+
+    def record_pre_step(self):


Missing return type

nv-caelan · 2025-08-28T21:46:53Z

scripts/imitation_learning/isaaclab_mimic/generate_dataset.py

+        # Cleanup of motion planners and their visualizers
+        if motion_planners is not None:
+            for env_id, planner in motion_planners.items():
+                if hasattr(planner, "plan_visualizer") and planner.plan_visualizer is not None:


Could instead do:

Suggested change

if hasattr(planner, "plan_visualizer") and planner.plan_visualizer is not None:

if getattr(planner, "plan_visualizer", None) is not None:

nv-caelan · 2025-08-28T21:48:56Z

source/isaaclab_mimic/isaaclab_mimic/datagen/datagen_info.py

@@ -30,6 +31,7 @@ def __init__(
        eef_pose=None,
        object_poses=None,
        subtask_term_signals=None,
+        subtask_start_signals=None,


Type hints for other args were declared in the function docstring as opposed to signature. Just following the same style.

nv-caelan · 2025-08-28T21:49:38Z

source/isaaclab_mimic/isaaclab_mimic/datagen/datagen_info_pool.py

+                        eef_subtask_boundaries[i - 1][1] + prev_max_offset_range
+                        < eef_subtask_boundaries[i][1] + self.subtask_term_offset_ranges[eef_name][i][0]
+                    ), (
+                        "subtask sanity check violation in demo with subtask {} end ind {}, subtask {} max offset {},"


Use f-strings

nv-caelan · 2025-08-28T22:20:43Z

source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/plan_visualizer.py

+
+        self._monitor_active = True
+
+        def monitor_parent_process():


Suggested change

def monitor_parent_process():

def monitor_parent_process() -> None

Added in the recent commit

nv-caelan · 2025-08-28T22:20:51Z

source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/plan_visualizer.py

+        self._monitor_thread = threading.Thread(target=monitor_parent_process, daemon=True)
+        self._monitor_thread.start()
+
+    def _kill_rerun_processes(self):


Suggested change

def _kill_rerun_processes(self):

def _kill_rerun_processes(self) -> None

Added in the recent commit

nv-caelan · 2025-08-28T22:22:53Z

.../isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py

+def cube_stack_test_env():
+    """Create the environment and motion planner once for the test suite and yield them."""
+    random.seed(SEED)
+    torch.manual_seed(SEED)


Check that you don't need to also set the numpy seed

Don't think so, will get back to this later

nv-caelan · 2025-08-28T22:23:27Z

.../isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py

+        marker_cfg.markers["frame"].scale = (0.1, 0.1, 0.1)
+        goal_pose_visualizer = VisualizationMarkers(marker_cfg)
+
+    yield {


Add a Generator return type?

nv-caelan · 2025-08-28T22:24:04Z

source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py

+    """Test suite for the Curobo motion planner, focusing on obstacle avoidance."""
+
+    @pytest.fixture(autouse=True)
+    def setup(self, curobo_test_env):


Return types here and elsewhere in TestCuroboPlanner

Added in the recent commit

peterd-NV

Tested/verified skillgen/prior mimic workflows working as expected. My prior review was completed in IsaacLab-Internal repo and all comments have been addressed in this public PR.

…emoving redundancies

njawale42 added 4 commits August 28, 2025 10:04

Porting SkillGen to the public repo

a690a40

A minor rerun related code update

7acaa43

Updated cuRobo and isaac sim licenses. Fixed rerun path viz bugs. Min…

cf32f5e

…or changes in cuRobo. Added urdf extraction and on-the-fly config creation from nucleus.

Added data gen test for skillgen, and changes to planner and visualizer

531582d

njawale42 requested review from jtigue-bdai, kellyguo11, Mayankm96 and ooctipus as code owners August 28, 2025 18:32

kellyguo11 requested a review from peterd-NV August 28, 2025 19:46

njawale42 requested review from peterd-NV and removed request for peterd-NV August 28, 2025 19:59

gavrielstate reviewed Aug 28, 2025

View reviewed changes

Minor change in README regarding isaacsim license file

ebca0ae

nv-caelan suggested changes Aug 28, 2025

View reviewed changes

njawale42 requested a review from nv-caelan August 28, 2025 22:36

peterd-NV approved these changes Aug 29, 2025

View reviewed changes

Addressing review comments: type hints, conditional simplification, r…

ada1a3a

…emoving redundancies

		@@ -188,6 +188,10 @@ where creativity and technology intersect. Your contributions can make a meaning

		The Isaac Lab framework is released under [BSD-3 License](LICENSE). The `isaaclab_mimic` extension and its corresponding standalone scripts are released under [Apache 2.0](LICENSE-mimic). The license files of its dependencies and assets are present in the [`docs/licenses`](docs/licenses) directory.

		Note that Isaac Lab requires Isaac Sim, which includes components under proprietary licensing terms. Please see the [Isaac Sim license](https://github.com/isaac-sim/IsaacSim/blob/main/LICENSE) for information on Isaac Sim licensing.

	if hasattr(planner, "plan_visualizer") and planner.plan_visualizer is not None:
	if getattr(planner, "plan_visualizer", None) is not None:

	def monitor_parent_process():
	def monitor_parent_process() -> None

	def _kill_rerun_processes(self):
	def _kill_rerun_processes(self) -> None

SkillGen in Isaac Lab for 2.3 Release #3303

Are you sure you want to change the base?

SkillGen in Isaac Lab for 2.3 Release #3303

Conversation

njawale42 commented Aug 28, 2025

Description

Technical Implementation:

Dependencies:

Integration:

Type of change

Screenshot

SkillGen Data Generation

Bin Cube Stacking Behavior Cloned Policy

Rerun Integration

Motion Planner Tests

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterd-NV left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

peterd-NV left a comment •

edited

Loading