[Feature]: Improve simulation UI for parameter tuning and replay support

[GaoLF]

What feature or enhancement are you proposing?

Hi, thanks for the great work!

Currently, the simulation UI lacks convenient tools for interactive parameter tuning. Every time we need to adjust parameters (e.g. constraint stiffness, damping, or initial velocity), we have to recompile and rerun the simulation from scratch. This can be time-consuming.

Motivation

The current workflow in Genesis for cloth simulation and robot-cloth interaction requires frequent recompilation when tuning simulation parameters. This slows down the iteration process, especially when experimenting with complex interactions such as grasping, folding, or regrasping cloth using robotic manipulators.

As cloth dynamics are highly sensitive to parameter choices (e.g., constraint stiffness, compliance, damping), a responsive, interactive UI would significantly improve development efficiency and simulation stability testing.

Potential Benefit

Faster iteration cycle: Adjust simulation parameters and immediately observe the effects, without recompiling.

Improved reproducibility: Enable controlled replay of interactions

Better usability

What is the expected outcome of the implementation work?

A lightweight UI panel to tweak common simulation parameters without recompiling.

Replay support or a simulation reset button to restart the simulation from frame 0 using the current parameters and mesh.

Optional: Hot reload of mesh files or config files (e.g. .json / .yaml) to make iteration smoother.

Additional information

No response

[Mehak261124]

Hey @duburcqa ,

Before beginning any implementation work on this feature, I wanted to share some findings from my exploration of the codebase to ensure alignment with the team's technical direction.

Genesis already provides a well-structured viewer plugin architecture (ViewerPlugin base class in genesis/vis/viewer_plugins/), with DefaultControlsPlugin and MouseInteractionPlugin serving as reference implementations. This system appears to be the most appropriate foundation for implementing interactive parameter tuning, as a dedicated plugin could hook into existing keyboard and mouse event handlers to modify simulation parameters at runtime — eliminating the need for recompilation.

Before proceeding, I would appreciate clarification on the following points:

1. UI approach — The current viewer supports user feedback via set_message_text(). For parameter tuning, would the team prefer a keyboard-driven interface (e.g. +/- keys to increment parameter values), or would integrating a lightweight panel library such as imgui be an acceptable direction?

2. Simulation reset — Is there an existing state serialization or checkpoint mechanism that could support rewinding the simulation to frame 0, or would this need to be designed from scratch?

3. Scope — What would be the recommended starting point for an initial contribution? A minimal keyboard-driven parameter plugin, or is there a specific component the team considers the highest priority?

[duburcqa]

@Mehak261124 You should probably have a look to this draft PR: #2541
This would give you an idea of what kind of GUI is being worked on internally. Note that the implementation in this PR is not satisfactory yet, and there is no support for remote WebRTC. You should also have a look to this PR: #2439 that was closed because of many design flaws and lack of proper unit testing. In both cases, I recommend that you read not only the PR but also the closed comments.

I think there are at least 3 important tools that we are lacking today:

• GUI for parameter tuning
• GUI for motion planning / animation
• GUI for scene crafting

I don't know if all these use cases can be unified as a single GUI. I think it is doable but it could make the task significantly more challenging. Note that today we already have a very basic entry point called gs view. It can be used to load and display any kind of assets, and control the joints configuration using sliders. We are not happy with it, but it is a start.

Regarding you questions more specifically:

1. A lightweight panel library such as imgui is probably unavoidable because there are too many parameters. Some options are defined at solver level, some are entity specific, some are link specific, some are geometry specific...etc Using keyboard for this would be a pain, unless it is used to prompt some AI model or something behind the scene that can make sense of unstructured request, but this is probably out of scope here.
2. You can reset the scene by calling scene.reset. You can also dump a state and restore it using scene.get_state and passing this output to scene.reset. But replay would require something else. For replay, you should store yourself the state of the simulation at every timestep and restore at every timestep. I don't think scene.reset is the correct API for this, it was not designed for replay. One option is to restrict this work to rigid body dynamics, and store scene.rigid_solver.get_state(), then call scene.rigid_solver.set_state(), then call scene.sim._sensor_manager.step()... Not great. Ideally we should add some proper API for generating log files and replay them. Internally, we have some recording system, but it is not exposed to the user and it was only used as a way to record sensor data at the moment, not to generate standalone simulation log files. Yet, having this would be very useful. Otherwise, one other viable and simpler option would be to just recording the camera frame at every timestep and just rewind them during replay. The issue with this is that it does not support moving the camera, which is fairly problematic because you looses half of the benefit of replay. Alternatively, you could dump the pose of every geometry in the scene, and the vertices for dynamically morphing geometries such as cloths or particle based meshes. This would work, but it is still limited because it cannot be used outside of scene replay. A full fledge logging and replaying mechanism is much more appealing.
3. The highest priority for Genesis' Team is probably a scene crafting GUI. Basically everything you need to conveniently generate a scene: the idea is to place objects in the scene using the mouse, using the physics to make sure it is a valid state, without penetrations or floating objects. The existing MouseInteraction plugin could be involved. Then have some export button to export the scene, either as MJCF, USD or Python to be decided. As a Python script is probably the most portable options but probably more tedious to implement. However, it should be quite possible now that all our option classes are clean and based on Pydantic that exposes proper serialisation and loading mechanism.