Add Haply device API with force feedback and teleoperation demo #3873

mingxueg-nv · 2025-10-29T07:49:12Z

Description

Add Haply haptic device teleoperation support for robotic manipulation with force feedback.

Type of change

New feature (non-breaking change which adds functionality)

Add haply devices API for teleoperation with force feedback
Real-time teleoperation with force feedback demo via Haply Inverse3 handle

Usage (make sure your Haply device is connected):

./isaaclab.sh -p scripts/demos/haply_teleoperation.py --websocket_uri ws://localhost:10001 --pos_sensitivity 1.65

Screenshots

Checklist

I have read and understood the contribution guidelines
I have run the pre-commit checks with ./isaaclab.sh --format
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
I have updated the changelog and the corresponding version in the extension's config/extension.toml file
I have added my name to the CONTRIBUTORS.md or my name already exists there

greptile-apps

Greptile Overview

Greptile Summary

This PR introduces support for Haply haptic devices to Isaac Lab's teleoperation ecosystem, enabling 6-DOF robot control with bidirectional force feedback. The implementation adds a new HaplyDevice class following the existing DeviceBase pattern used by keyboard, gamepad, and spacemouse interfaces. The device communicates with Haply SDK via WebSocket in a separate thread to receive position/orientation from Inverse3 and VerseGrip hardware while sending force feedback commands. A standalone demo script (haply_teleoperation.py) demonstrates real-time teleoperation of a Franka Panda robot using differential IK control, mapping the Haply workspace to robot end-effector poses with contact-based force feedback. The changes integrate cleanly into the existing device factory pattern without modifying core teleoperation infrastructure.

Important Files Changed

Filename	Score	Overview
source/isaaclab/isaaclab/devices/init.py	5/5	Adds Haply device documentation and imports following existing device pattern
source/isaaclab/isaaclab/devices/haply/init.py	5/5	Creates new Haply module initialization exporting HaplyDevice and HaplyDeviceCfg
source/isaaclab/isaaclab/devices/teleop_device_factory.py	5/5	Registers HaplyDeviceCfg in device factory mapping for instantiation
source/isaaclab/isaaclab/devices/haply/se3_haply.py	3.5/5	Implements core Haply device controller with WebSocket communication and force feedback (hardcoded availability flag, potential connection logic issue)
scripts/demos/haply_teleoperation.py	3/5	Adds teleoperation demo with Franka Panda (hardcoded transforms, potential joint index conflict at line 318-320, force feedback always computed but conditionally applied)

Confidence score: 3/5

This PR requires careful review due to implementation issues in core device logic and the demo script
Score lowered due to: (1) hardcoded WEBSOCKETS_AVAILABLE flag that doesn't respect import failures in se3_haply.py, (2) connection logic using OR condition may allow operation with incomplete device setup, (3) potential joint index conflict in haply_teleoperation.py where joint 7 (index 6) could be set twice based on arm DOF, (4) hardcoded workspace mappings without validation, (5) no graceful WebSocket thread shutdown mechanism
Pay close attention to source/isaaclab/isaaclab/devices/haply/se3_haply.py (lines 18-20, 81-82, 264-266, 292-294) and scripts/demos/haply_teleoperation.py (lines211, 318-320) for potential runtime issues

Sequence Diagram

sequenceDiagram
    participant User
    participant HaplyDevice
    participant WebSocket
    participant SimulationContext
    participant Robot
    participant IKController
    participant ContactSensor

    User->>HaplyDevice: Initialize connection (websocket_uri)
    HaplyDevice->>WebSocket: Connect to Haply SDK
    WebSocket-->>HaplyDevice: Connection established
    HaplyDevice->>HaplyDevice: Start background thread
    
    User->>SimulationContext: Start simulation
    SimulationContext->>Robot: Initialize Franka Panda
    SimulationContext->>ContactSensor: Setup finger sensors
    
    loop Simulation Loop
        User->>HaplyDevice: Move handle / Press buttons
        HaplyDevice->>WebSocket: Request device state
        WebSocket-->>HaplyDevice: Return position, orientation, buttons
        HaplyDevice->>HaplyDevice: Cache data with lock
        
        SimulationContext->>HaplyDevice: advance()
        HaplyDevice-->>SimulationContext: Return [pos, quat, buttons]
        
        SimulationContext->>SimulationContext: apply_haply_to_robot_mapping()
        SimulationContext->>IKController: compute(target_pos, target_quat)
        IKController-->>SimulationContext: Return joint_pos_des
        
        SimulationContext->>Robot: set_joint_position_target()
        Robot->>Robot: Update joint positions
        
        alt Force Feedback Enabled
            ContactSensor->>ContactSensor: Measure net_forces_w
            SimulationContext->>ContactSensor: Get finger contact forces
            ContactSensor-->>SimulationContext: Return force vectors
            SimulationContext->>SimulationContext: Calculate feedback_force
            SimulationContext->>HaplyDevice: set_force_feedback(fx, fy, fz)
            HaplyDevice->>WebSocket: Send force command
            WebSocket-->>User: Apply haptic feedback
        end
        
        SimulationContext->>SimulationContext: step()
    end
    
    User->>SimulationContext: Close application
    SimulationContext->>HaplyDevice: Cleanup
    HaplyDevice->>WebSocket: Disconnect

_{5 files reviewed, 6 comments}

_{Edit Code Review Agent Settings | Greptile}

scripts/demos/haply_teleoperation.py

source/isaaclab/isaaclab/devices/haply/se3_haply.py

greptile-apps

Greptile Overview

Greptile Summary

This review covers only the changes made since the last review, not the entire PR. The latest changes involve two focused updates: (1) removal of a quaternion conversion comment from the teleoperation demo script, and (2) comprehensive removal of all inline comments and section dividers from the HaplyDevice class implementation. The comment removal in se3_haply.py reduces code verbosity by eliminating explanatory text for operations like "Apply sensitivity scaling" and "Detect rising edge", while preserving all docstrings and functional logic. This makes the code more concise but relies entirely on method names and docstrings for understanding the WebSocket communication, threading, and force feedback implementation. The demo script change is minimal, removing a redundant comment about quaternion conversion. Both changes align with a code style preference for self-documenting code over inline annotations, though this may impact readability for contributors unfamiliar with haptic device protocols or the Haply WebSocket API.

Important Files Changed

Filename	Score	Overview
scripts/demos/haply_teleoperation.py	4/5	Removed quaternion conversion comment on line 117; purely cosmetic change with no logic modifications
source/isaaclab/isaaclab/devices/haply/se3_haply.py	4/5	Stripped all inline comments and section dividers while preserving docstrings; no functional changes but reduced code documentation

Confidence score: 4/5

This PR is safe to merge with minimal risk as the changes are purely cosmetic comment removals with no functional modifications
Score reflects that while the changes reduce inline documentation, all core logic, docstrings, and functionality remain intact; no new bugs are introduced by these changes
No files require special attention as both changes are straightforward comment removals; previous review concerns about logic issues (lines 317-319 data race) and performance (deep copying at 200Hz) remain unaddressed but are not affected by these latest changes

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

source/isaaclab/isaaclab/devices/haply/se3_haply.py

greptile-apps

Greptile Overview

Greptile Summary

This review covers only the changes made since the last review, not the entire PR. The most recent changes refactor the Haply teleoperation demo by removing the --enable_force_feedback CLI flag and making force feedback always-on. The se3_haply.py device API was enhanced with proper cleanup mechanisms, improved connection monitoring with timeout tracking, and optimized data processing by removing unnecessary deep copies. Key structural improvements include extracting workspace mapping constants to module-level scope, adding a close() method for graceful device shutdown with try-finally cleanup, and fixing the WebSocket message format to use proper list structure. The connection logic was changed from OR to AND semantics (requiring both Inverse3 and VerseGrip devices), which may affect partial-operation scenarios. The refactoring indicates maturation from experimental to production-ready feature status.

Important Files Changed

Filename	Score	Overview
scripts/demos/haply_teleoperation.py	3/5	Removes force feedback CLI flag, extracts workspace constants, adds try-finally cleanup; joint position overwrite issue on line 318-319 remains unresolved from previous review
source/isaaclab/isaaclab/devices/haply/se3_haply.py	4/5	Adds optional websockets import handling, implements `close()` method, improves connection monitoring with timeout tracking, removes deep copies in high-frequency loop; connection logic changed from OR to AND requiring both devices

Confidence score: 3/5

This PR has robustness improvements but contains an unresolved logic issue and a potentially breaking connection requirement change that needs verification
Score reflects: (1) the unresolved joint position overwrite issue in haply_teleoperation.py line 318-319 that was flagged in previous reviews but not fixed, which may discard IK-computed values for joint 7; (2) the connection logic change from OR to AND semantics in se3_haply.py line 275-277 that now requires BOTH Inverse3 and VerseGrip devices to be connected, which is a breaking change that may not be intentional and needs verification; (3) the PR description still references the removed --enable_force_feedback flag in the usage example, creating documentation inconsistency
Pay close attention to haply_teleoperation.py lines 318-320for the joint position assignment logic and se3_haply.py lines 275-277 for the device connection requirements

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

source/isaaclab/isaaclab/devices/haply/se3_haply.py

greptile-apps

Greptile Overview

Greptile Summary

This review covers only changes made since the last review, focusing on thread management and cleanup improvements in the Haply device WebSocket connection handling. The changes address previous feedback by adding explicit thread joining with timeout in the close() method and ensuring _websocket_thread is initialized before the thread starts to prevent race conditions. The WebSocket thread is now non-daemon to enable proper cleanup through join(), though this requires that close() is called explicitly to avoid hanging the process at exit.

Important Files Changed

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	3/5	Improved WebSocket thread cleanup with explicit join and timeout handling; changed thread to non-daemon which requires explicit cleanup to prevent hangs

Confidence score: 3/5

This PR addresses previous feedback on thread cleanup but introduces a potential process hang if close() is not called before exit
Score reflects improved cleanup logic offset by the non-daemon thread change requiring explicit cleanup, plus unaddressed thread safety concerns between advance() and reset() methods accessing shared _prev_buttons state without synchronization
Pay close attention to the thread joining logic in close() (lines 165-168) and verify that close() is reliably called in all exit paths; also verify thread safety of _prev_buttons access between advance() and reset() methods

_{1 file reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

Greptile Overview

Greptile Summary

This review covers only the changes made since the last review, not the entire PR. The latest update adds thread-safe button state tracking to the Haply haptic device teleoperation API. A new _button_lock threading lock was introduced to protect concurrent access to the _prev_buttons dictionary, which tracks button states for rising-edge detection. The lock is used in both the reset() method and the advance() method where button press events are detected. This addresses a race condition where the WebSocket thread (running _websocket_loop()) updates button states while the main thread reads them in advance() for edge detection. The implementation follows the existing synchronization pattern used for force_lock and data_lock in the same class. Additionally, the cleanup logic in __del__ was enhanced to warn about daemon thread conversion if graceful shutdown fails.

Important Files Changed

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	3/5	Added `_button_lock` for thread-safe button state tracking to prevent race conditions during rising-edge detection

Confidence score: 3/5

This PR requires careful review due to potential issues with the thread safety implementation and callback execution within locked sections
Score reflects concerns about holding locks during callback invocation (lines 236-245) which could cause deadlocks if callbacks take significant time or attempt to acquire the same lock, and the lack of explicit timeout or error handling for lock acquisition
Pay close attention to the button event detection logic in advance() method where callbacks are invoked while holding _button_lock, as this could block the WebSocket thread if callbacks are slow

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

source/isaaclab/isaaclab/devices/haply/se3_haply.py

greptile-apps

Greptile Overview

Greptile Summary

This review covers changes made since the last review, focusing on recent modifications to the Haply device controller. The refactor removes the connection_timeout parameter and time-based "data freshness" checks (get_device_state(), _is_data_fresh(), last_data_time) in favor of simpler binary connection state tracking. A critical logic change switches the connection check from OR to AND, now requiring both Inverse3 and VerseGrip devices to be connected simultaneously. Thread safety has been improved with a dedicated _connected_lock separating connection state from data synchronization, and the _button_lock was removed to allow callbacks to execute outside locks, addressing previous deadlock concerns. However, the refactor introduces race conditions where cached_data is accessed without holding data_lock, and array indexing operations lack bounds checking which could cause IndexErrors.

Important Files Changed

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	2/5	Refactored device connection management removing timeout logic, changed connection check from OR to AND, improved lock separation but introduced race conditions in cached_data access and missing bounds checks on array indexing

Confidence score: 2/5

This PR has significant thread-safety issues that could cause runtime failures or race conditions in production
Score reflects critical bugs: (1) cached_data accessed without data_lock at lines 331-334 during connection status updates, (2) Array indexing at lines 288-289 without bounds checking could raise IndexError if WebSocket returns empty arrays, (3) Connection status check at lines 206-217 is separated from data access creating a window where device could disconnect between checks
Pay close attention to lines 288-289 (array indexing), lines 331-334 (unlocked cached_data access), and lines 206-217 (connection check race condition). The AND logic change at line 332-333 should also be verified as intentional behavior change requiring both devices

_{1 file reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

source/isaaclab/isaaclab/devices/haply/se3_haply.py

greptile-apps

Greptile Overview

Greptile Summary

This review covers only the changes made since the last review, not the entire PR. The developer addressed critical race conditions and thread safety issues in the Haply device teleoperation implementation. The primary fix involves consolidating connection status checks under proper lock synchronization: the advance() method now reads connection state inside data_lock to prevent stale reads, while the WebSocket loop computes connection status locally before atomically updating both cached_data and the self.connected flag. Additional improvements include safer handling of empty device data arrays to prevent IndexError exceptions, and consistent connection state resets in exception handlers under appropriate locks. These changes maintain the existing two-lock design pattern while ensuring thread-safe state transitions between the WebSocket thread and the main simulation thread.

Important Files Changed

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	4/5	Fixed race condition by moving connection checks inside data_lock; improved empty list handling; made connection status updates atomic

Confidence score: 4/5

This PR is safer to merge than previous versions but still has some residual threading concerns that warrant attention.
Score reflects successful resolution of the critical race condition in connection status checking, proper synchronization of state updates, and improved exception safety. However, one point deducted because the callback invocation inside _button_lock (raised in previous reviews) remains unaddressed, which could potentially block the simulation loop or cause deadlocks if callbacks acquire other locks or perform time-consuming operations.
Pay close attention to the button callback logic within the lock in _update_button_callbacks() - consider testing scenarios where button callbacks interact with other locked resources or take significant execution time.

_{1 file reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

Greptile Overview

Greptile Summary

Adds Haply haptic device (Inverse3 + VerseGrip) API and teleoperation demo for robotic manipulation with bidirectional force feedback.

Key Changes:

New HaplyDevice class in isaaclab/devices/haply/ with WebSocket-based communication (200Hz)
Teleoperation demo showing Franka Panda control with IK, gripper control, and force feedback
Integration into device factory pattern for seamless device switching
Position/orientation tracking from hardware devices with button callbacks

Issues Found:

Critical race condition in se3_haply.py:207-214: advance() reads cached_data without holding data_lock, causing potential data corruption when WebSocket thread updates the same data
Callback deadlock risk in se3_haply.py:217-224: Button callbacks invoked while holding _button_lock (though current code doesn't use _button_lock, the pattern in _prev_buttons update could block simulation if callbacks are slow)
IndexError risk in se3_haply.py:287-288: Empty device arrays from SDK will raise IndexError on [0] access
Breaking compatibility change in se3_haply.py:334: Connection logic changed from OR to AND, now requires both devices instead of allowing partial operation
Minor: Joint assignment pattern in demo overwrites IK solution for joint 7

Confidence Score: 2/5

Not safe to merge - contains critical race condition that can cause data corruption in multi-threaded environment
Score of 2 reflects the race condition in advance() method accessing cached_data without proper locking (lines 207-214 in se3_haply.py). This is a critical thread-safety issue where the main simulation thread reads device data while the WebSocket thread concurrently updates it, potentially causing torn reads or inconsistent state. The IndexError risk on empty arrays and the breaking compatibility change (OR→AND) are also concerning.
source/isaaclab/isaaclab/devices/haply/se3_haply.py requires immediate attention for race condition fix. The advance() method must acquire data_lock before reading cached_data.

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	2/5	Core Haply device implementation with several critical issues: race condition in advance() accessing cached_data without lock (line 207-214), callback deadlock risk with button state updates happening inside lock (line 217-224), IndexError risk on empty device arrays (line 287-288), and changed device connection logic from OR to AND breaking backwards compatibility (line 334)
scripts/demos/haply_teleoperation.py	3/5	Teleoperation demo with IK-based control and force feedback. Contains joint index mismatch where joint 7 (index 6) is set manually after IK solution includes it (line 318-319), and force feedback computed unconditionally every iteration (line 332-336)

Sequence Diagram

sequenceDiagram
    participant User
    participant Demo as haply_teleoperation.py
    participant Device as HaplyDevice
    participant WS as WebSocket Thread
    participant SDK as Haply SDK
    participant Robot as Franka Robot
    participant Sensors as Contact Sensors

    User->>Demo: Start teleoperation demo
    Demo->>Device: HaplyDevice(cfg)
    Device->>WS: Start WebSocket thread
    WS->>SDK: Connect via WebSocket
    SDK-->>WS: Connection established
    WS->>SDK: Receive device data (200Hz)
    SDK-->>WS: {inverse3: [], wireless_verse_grip: []}
    WS->>Device: Update cached_data (position, orientation, buttons)
    
    loop Simulation Loop (200Hz)
        Demo->>Device: advance()
        Device->>Device: Read cached_data (potential race condition)
        Device->>Device: Check button callbacks
        Device-->>Demo: [x,y,z, qx,qy,qz,qw, btn_a, btn_b, btn_c]
        Demo->>Demo: Apply coordinate mapping
        Demo->>Demo: Compute IK for arm joints
        Demo->>Robot: set_joint_position_target()
        Robot->>Sensors: Measure contact forces
        Sensors-->>Demo: net_forces_w
        Demo->>Device: set_force_feedback(fx, fy, fz)
        Device->>WS: Update feedback_force
        WS->>SDK: Send force command via WebSocket
        SDK-->>WS: Acknowledgment
    end

    User->>Demo: Stop simulation
    Demo->>Device: close()
    Device->>WS: running = False
    WS->>SDK: Close WebSocket connection
    Device->>Device: Wait for thread termination

_{5 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

source/isaaclab/isaaclab/devices/haply/se3_haply.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: mingxueg-nv <mingxueg@nvidia.com>

greptile-apps

Greptile Overview

Greptile Summary

Added comprehensive Haply device API for teleoperation with force feedback. The implementation provides WebSocket-based communication with Haply Inverse3 and VerseGrip devices, enabling real-time position tracking, orientation sensing, button input, and bidirectional force feedback at 200Hz.

Key changes:

New HaplyDevice class with thread-safe data handling using multiple locks
WebSocket client running in dedicated thread for async communication
Device connection monitoring with timeout detection and auto-reconnection
Force feedback support sent to Inverse3 device
Button event callbacks with rising edge detection
Proper cleanup and thread termination in close() method

Latest commit (243b8ae) addresses critical race condition by adding .copy() calls inside the data lock when reading position and quaternion arrays, preventing potential data corruption from concurrent access.

Confidence Score: 4/5

Safe to merge with minor style improvements recommended
The latest commit fixed the critical race condition identified in previous review. Thread safety mechanisms are properly implemented with appropriate locks. Connection handling and error recovery are robust. Remaining comments are style/performance suggestions that don't affect correctness.
No files require special attention - the critical race condition has been resolved

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	4/5	New Haply device API with WebSocket communication, force feedback, and thread-safe data handling. Latest commit fixes critical race condition.

Sequence Diagram

sequenceDiagram
    participant App as Application
    participant HD as HaplyDevice
    participant WSThread as WebSocket Thread
    participant HaplySDK as Haply SDK Server

    App->>HD: __init__(cfg)
    HD->>HD: Initialize locks & data structures
    HD->>WSThread: Start WebSocket thread
    WSThread->>HaplySDK: Connect (WebSocket)
    
    loop Every 1/data_rate seconds (200Hz default)
        HaplySDK->>WSThread: Send device state (JSON)
        WSThread->>WSThread: Parse position, quaternion, buttons
        WSThread->>HD: Update cached_data (with data_lock)
        WSThread->>HD: Update connection status
        WSThread->>HaplySDK: Send force feedback (with force_lock)
    end
    
    App->>HD: advance()
    HD->>HD: Check connection status
    HD->>HD: Copy position, quaternion, buttons (with data_lock)
    HD->>HD: Trigger button callbacks if pressed
    HD->>App: Return torch.Tensor[10]
    
    App->>HD: set_force_feedback(fx, fy, fz)
    HD->>HD: Update feedback_force (with force_lock)
    
    App->>HD: close()
    HD->>HD: Set running=False, reset force
    HD->>WSThread: Wait for thread termination
    WSThread->>HaplySDK: Disconnect

_{1 file reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

Greptile Overview

Greptile Summary

This PR adds comprehensive Haply haptic device support for robotic teleoperation with bidirectional force feedback. The implementation includes a WebSocket-based device controller (se3_haply.py) that connects to Haply Inverse3 (position tracking) and VerseGrip (orientation/buttons) devices, plus a complete teleoperation demo (haply_teleoperation.py) showing Franka Panda arm control with contact-based force feedback.

Key additions:

HaplyDevice class with async WebSocket communication, thread-safe data handling, and force feedback API
Teleoperation demo with IK-based position control, workspace mapping, and contact sensor force feedback
Comprehensive test coverage with proper mocking for device constructors

Issues identified:

Critical joint indexing bug in demo (line 317): IK computes 6 joints but [:-1] slice incorrectly handles assignment
Previous comments highlighted race conditions in se3_haply.py (data_lock usage, button callback timing) and connection logic issues

Confidence Score: 2/5

This PR has critical bugs that will cause incorrect robot motion and previous unresolved race conditions
Score reflects one new critical logic error in joint assignment (line 317) that will cause incorrect IK joint application, plus multiple unresolved issues from previous review including race conditions in se3_haply.py that could cause thread safety problems and data corruption. The test coverage is excellent but doesn't catch the runtime logic bug.
scripts/demos/haply_teleoperation.py requires immediate fix to joint indexing logic before merge

Important Files Changed

File Analysis

Filename	Score	Overview
scripts/demos/haply_teleoperation.py	3/5	New teleoperation demo with Haply device - has joint indexing issue where IK output for 6 joints is assigned but joint 7 is also referenced causing potential mismatch
source/isaaclab/test/devices/test_device_constructors.py	5/5	Comprehensive test coverage added for Haply device with proper mocking of WebSocket connections and device states

Sequence Diagram

sequenceDiagram
    participant User as User/Haply Device
    participant WS as WebSocket Thread
    participant Device as HaplyDevice
    participant Demo as haply_teleoperation.py
    participant IK as IK Controller
    participant Robot as Franka Robot
    participant Sensor as Contact Sensors
    
    Note over User,WS: Device Initialization
    User->>WS: Connect via WebSocket
    WS->>Device: Receive position (Inverse3)
    WS->>Device: Receive orientation & buttons (VerseGrip)
    Device->>Device: Cache data with lock
    
    Note over Demo,Robot: Teleoperation Loop
    Demo->>Device: advance()
    Device->>Demo: Return [pos, quat, buttons]
    Demo->>Demo: Map Haply workspace to robot frame
    Demo->>IK: set_command(target_pos, ee_quat)
    IK->>Demo: Compute joint positions (6 joints)
    Demo->>Robot: set_joint_position_target()
    Robot->>Robot: Execute motion (5 sim steps)
    
    Note over Sensor,Device: Force Feedback Path
    Robot->>Sensor: Contact forces on fingers
    Sensor->>Demo: Read net_forces_w
    Demo->>Demo: Compute feedback_force (clamped)
    Demo->>Device: set_force_feedback(fx, fy, fz)
    Device->>Device: Store in feedback_force with lock
    WS->>Device: Get feedback_force
    WS->>User: Send cursor_force command to Inverse3

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

scripts/demos/haply_teleoperation.py

greptile-apps

Greptile Overview

Greptile Summary

Adds Haply haptic device teleoperation support with bidirectional force feedback for robotic manipulation. The implementation includes a WebSocket-based device API (HaplyDevice) and a complete demo script for controlling a Franka Panda arm.

Key Changes:

New HaplyDevice API providing position tracking (Inverse3), orientation/buttons (VerseGrip), and force feedback
Teleoperation demo with IK control for 6-DOF arm positioning, button-based gripper/rotation control, and contact-based haptic feedback
Thread-safe WebSocket communication with proper connection monitoring and auto-reconnect
Workspace safety limits and coordinate mapping from Haply to robot frame

Recent Fix:

Latest commit (13c4047) correctly fixed IK joint index issue - now properly assigns all 6 IK-computed joints to panda_joint1-6 via arm_joint_indices

Minor Issues:

VerseGrip orientation data computed but unused (intentional design for button-based rotation)
Python list vs numpy array inconsistency for haply_initial_pos
Previous comments cover threading, race conditions, and error handling in device API

Confidence Score: 4/5

This PR is safe to merge with minor style improvements recommended
The recent commit fixed the critical IK joint index issue. The implementation is functionally correct with proper error handling, thread safety in the device API, and workspace safety limits. Previous review comments identified race conditions and threading issues in the device API that should be addressed, but the demo script itself is solid. The unused orientation data is a design choice rather than a bug.
No files require special attention - the IK index bug was already fixed in commit 13c4047

Important Files Changed

File Analysis

Filename	Score	Overview
scripts/demos/haply_teleoperation.py	4/5	Teleoperation demo correctly fixed IK joint index issue; orientation from VerseGrip unused but this appears intentional for button-based rotation control

Sequence Diagram

sequenceDiagram
    participant User
    participant HaplyDevice
    participant WebSocket
    participant Simulation
    participant Robot
    participant IKController
    participant ContactSensor

    User->>HaplyDevice: Initialize(websocket_uri, pos_sensitivity)
    HaplyDevice->>WebSocket: Connect to Haply SDK
    WebSocket-->>HaplyDevice: Connection established
    HaplyDevice->>HaplyDevice: Start WebSocket thread
    WebSocket-->>HaplyDevice: Device data (Inverse3 + VerseGrip)
    HaplyDevice-->>User: Device ready

    loop Teleoperation Loop
        Simulation->>HaplyDevice: advance()
        HaplyDevice-->>Simulation: [pos, quat, buttons]
        
        Simulation->>Simulation: apply_haply_to_robot_mapping()
        Simulation->>Simulation: Handle button events (gripper, rotation)
        
        Simulation->>IKController: set_command(target_pos, ee_quat)
        IKController->>Robot: Get current_joint_pos, jacobian
        Robot-->>IKController: Joint states
        IKController-->>Simulation: joint_pos_des (6 joints)
        
        Simulation->>Robot: set_joint_position_target()
        
        loop 5 simulation steps
            Simulation->>Simulation: write_data_to_sim()
            Simulation->>Simulation: step()
        end
        
        Simulation->>Simulation: update(sim_dt)
        
        Simulation->>ContactSensor: Get finger forces
        ContactSensor-->>Simulation: net_forces_w
        Simulation->>Simulation: Calculate feedback_force
        Simulation->>HaplyDevice: set_force_feedback(fx, fy, fz)
        
        HaplyDevice->>WebSocket: Send force command to Inverse3
        WebSocket-->>User: Apply haptic feedback
    end

_{1 file reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

scripts/demos/haply_teleoperation.py

source/isaaclab/isaaclab/devices/haply/se3_haply.py

ooctipus · 2025-11-03T19:39:13Z

@rwiltz
Yes I think that could work, you could also add the torque into the signature as well if you'd like: ))

greptile-apps

Greptile Overview

Greptile Summary

Adds comprehensive Haply haptic device support for robotic teleoperation with force feedback. The implementation integrates Haply Inverse3 (position tracking + force feedback) and VerseGrip (orientation + buttons) via WebSocket communication.

Key Components:

HaplyDevice class with thread-safe WebSocket communication at 200Hz
Teleoperation demo with IK-based arm control, button-driven gripper/rotation, and bidirectional force feedback
Comprehensive unit tests with mocked WebSocket/threading infrastructure
Complete documentation including hardware requirements and setup guide

Recent Fixes (commit 9105f73):

Resolved critical race condition by moving .copy() calls inside data_lock in advance() method
Removed invalid orientation_sensitivity parameter (quaternions must remain normalized)
Fixed IndexError risk in WebSocket loop with proper empty list handling
Added boundary checks in push_force() for empty force arrays

Architecture:
The integration follows Isaac Lab's device abstraction pattern, extending DeviceBase and adding push_force() method for haptic feedback. The demo showcases coordinate mapping from Haply workspace to robot workspace with workspace limits for safety.

Confidence Score: 4/5

This PR is safe to merge after addressing the thread cleanup concern - the core functionality is solid with proper thread safety after recent fixes
Score reflects that all critical threading and race condition issues were resolved in commit 9105f73. The implementation demonstrates good thread safety patterns (locks for shared data, callbacks outside locks), comprehensive testing, and thorough documentation. Minor deduction for thread cleanup behavior in __del__ which could be improved but is not critical.
source/isaaclab/isaaclab/devices/haply/se3_haply.py - Review the __del__ thread cleanup logic (lines 144-162) to ensure graceful shutdown

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	4/5	Implements Haply haptic device interface with WebSocket communication, thread-safe data handling, and force feedback. Recent fixes addressed critical race conditions by moving `.copy()` inside locks.
scripts/demos/haply_teleoperation.py	5/5	Teleoperation demo using Haply device with IK-based control. Joint assignment logic is correct: 6 joints from IK (panda_joint1-6), joint7 from button control, gripper from buttons A/B.
source/isaaclab/test/devices/test_device_constructors.py	5/5	Comprehensive unit tests for Haply device with mocked WebSocket, threading, and asyncio. Tests constructor, force feedback, reset, and connection timeout scenarios.
source/isaaclab/isaaclab/devices/device_base.py	5/5	Added `push_force()` method to base class as optional no-op, allowing devices with haptic feedback (like Haply) to override while maintaining compatibility with non-haptic devices.

Sequence Diagram

sequenceDiagram
    participant User as Operator
    participant Demo as haply_teleoperation.py
    participant Haply as HaplyDevice
    participant WS as WebSocket Thread
    participant SDK as Haply SDK
    participant Robot as Robot (Isaac Sim)
    participant IK as IK Controller
    participant Sensor as Contact Sensors

    User->>SDK: Move Inverse3 handle
    User->>SDK: Press VerseGrip buttons
    SDK->>WS: WebSocket data (200Hz)
    WS->>Haply: Update cached_data (with data_lock)
    
    Demo->>Haply: advance()
    Haply->>Haply: Lock data_lock
    Haply->>Haply: Copy position, quaternion, buttons
    Haply->>Haply: Unlock data_lock
    Haply->>Haply: Execute button callbacks (outside lock)
    Haply-->>Demo: Return [pos(3), quat(4), buttons(3)]
    
    Demo->>Demo: Map Haply → Robot workspace
    Demo->>Demo: Handle button A/B (gripper)
    Demo->>Demo: Handle button C (EE rotation)
    Demo->>IK: Compute joint targets (6 joints)
    IK-->>Demo: Joint positions
    Demo->>Robot: Set joint targets (6 IK + 1 rotation + 2 gripper)
    
    Robot->>Sensor: Update contact forces
    Sensor-->>Demo: Net forces from fingers
    Demo->>Haply: push_force(contact_forces)
    Haply->>Haply: Lock force_lock
    Haply->>Haply: Clip and store feedback_force
    Haply->>Haply: Unlock force_lock
    Haply->>WS: Send force command
    WS->>SDK: WebSocket force feedback
    SDK->>User: Apply haptic feedback

_{13 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

Greptile Overview

Greptile Summary

This PR adds comprehensive Haply haptic device support for teleoperation with force feedback. The latest commit (bd78c0f) adjusts workspace limits and table positioning to improve the demo setup.

Key Changes:

Workspace x-limit expanded from (0.2, 0.9) to (0.1, 0.9) in scripts/demos/haply_teleoperation.py:76
Robot initial position moved from (0.1, 0.0, 1.05) to (-0.02, 0.0, 1.05) in scripts/demos/haply_teleoperation.py:144
Table rotation added: rot=(0.707, 0, 0, 0.707) and scale changed from (2.0, 1.0, 1.0) to (1.0, 1.0, 1.0) in scripts/demos/haply_teleoperation.py:140
Cube position adjusted from (0.65, -0.05, 1.15) to (0.60, 0.00, 1.15) in scripts/demos/haply_teleoperation.py:157
Updated screenshot with smaller file size (536KB → 98KB)

Previous Review Findings:
Multiple critical threading and concurrency issues were identified in earlier commits and have been addressed in commit 9105f73. The current head commit only makes minor positioning adjustments and does not introduce new issues.

Confidence Score: 4/5

This PR is safe to merge with minor configuration changes that improve demo positioning
The latest commit only adjusts positioning parameters (workspace limits, robot/table/cube positions) which are low-risk configuration changes. Previous critical threading and concurrency issues in the core se3_haply.py were addressed in earlier commits. The demo script follows established patterns for IK control and teleoperation.
No files require special attention - changes are cosmetic positioning adjustments

Important Files Changed

File Analysis

Filename	Score	Overview
scripts/demos/haply_teleoperation.py	4/5	New demo script implementing Haply teleoperation for Franka robot with IK control, force feedback, and button-based gripper/rotation control. Latest commit adjusts workspace limits and table positioning.
docs/source/_static/demos/haply_teleop_franka.jpg	5/5	Updated screenshot image for documentation - binary file change with reduced file size (536KB to 98KB).

Sequence Diagram

sequenceDiagram
    participant User
    participant Demo as haply_teleoperation.py
    participant Haply as HaplyDevice
    participant WS as WebSocket Thread
    participant SDK as Haply SDK
    participant Robot as Franka Robot
    participant Sensors as Contact Sensors

    User->>Demo: Launch demo script
    Demo->>Haply: Initialize HaplyDevice(cfg)
    Haply->>WS: Start websocket_thread
    WS->>SDK: Connect via WebSocket
    SDK-->>WS: Device data stream
    WS->>Haply: Update cached_data (position, orientation, buttons)
    Haply-->>Demo: Device ready

    loop Simulation Loop
        Demo->>Haply: advance() - get device state
        Haply-->>Demo: [x,y,z, qx,qy,qz,qw, btn_a, btn_b, btn_c]
        Demo->>Demo: apply_haply_to_robot_mapping()
        Demo->>Demo: IK compute() for joints 1-6
        Demo->>Demo: Button C controls joint 7 rotation
        Demo->>Demo: Buttons A/B control gripper
        Demo->>Robot: set_joint_position_target()
        Robot->>Sensors: Contact forces
        Sensors-->>Demo: net_forces_w (left + right fingers)
        Demo->>Haply: push_force(total_contact_force)
        Haply->>WS: Update feedback_force
        WS->>SDK: Send force command via WebSocket
        SDK->>User: Apply force feedback to Inverse3
    end

_{1 file reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

Greptile Overview

Greptile Summary

This PR adds a binary image file (haply_teleop_franka.jpg) to the documentation assets directory. This is part of the larger Haply device teleoperation feature that was added in previous commits.

The image file is used to illustrate the Haply teleoperation demo in the documentation and showroom pages.

Confidence Score: 5/5

This PR is completely safe to merge - it only adds a documentation image file with no code changes
This PR only adds a single binary image file for documentation purposes. There are no code changes, no logic modifications, and no potential runtime issues. The image is used to illustrate the Haply teleoperation feature in the documentation.
No files require special attention - this is a simple documentation image addition

Important Files Changed

File Analysis

Filename	Score	Overview
docs/source/_static/demos/haply_teleop_franka.jpg	5/5	Binary image file added for documentation - safe, no logic changes

Sequence Diagram

sequenceDiagram
    participant User
    participant HaplySDK
    participant HaplyDevice
    participant Simulator
    participant Robot
    participant ContactSensor

    User->>HaplySDK: Move Inverse3 handle
    User->>HaplySDK: Press VerseGrip buttons
    
    loop Every 200Hz
        HaplySDK->>HaplyDevice: WebSocket: position, orientation, buttons
        HaplyDevice->>HaplyDevice: Update cached_data (thread-safe)
        
        Simulator->>HaplyDevice: advance()
        HaplyDevice->>HaplyDevice: Copy position, quaternion, buttons
        HaplyDevice-->>Simulator: [x,y,z, qx,qy,qz,qw, btn_a,btn_b,btn_c]
        
        Simulator->>Simulator: Map Haply pos to robot workspace
        Simulator->>Simulator: Compute IK for 6 arm joints
        Simulator->>Simulator: Set joint7 rotation (Button C)
        Simulator->>Simulator: Set gripper (Buttons A/B)
        
        Simulator->>Robot: Set joint position targets
        Robot->>Robot: Execute motion (5 physics steps)
        
        Robot->>ContactSensor: Update contact forces
        ContactSensor-->>Simulator: net_forces_w
        
        Simulator->>HaplyDevice: push_force(contact_forces)
        HaplyDevice->>HaplyDevice: Clip forces to safety limits
        HaplyDevice->>HaplySDK: WebSocket: set_cursor_force
        HaplySDK->>User: Apply force feedback to handle
    end

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

Greptile Overview

Greptile Summary

This PR adds comprehensive Haply haptic device support for Isaac Lab teleoperation with bidirectional force feedback. The implementation includes a new HaplyDevice class that interfaces with Haply Inverse3 (position tracking) and VerseGrip (orientation/buttons) via WebSocket, plus a complete teleoperation demo.

Key changes:

Added HaplyDevice class with WebSocket communication (200Hz), thread-safe data exchange, and force feedback limited to 2.0N for safety
Extended DeviceBase with push_force() method for haptic feedback support
Implemented teleoperation demo with IK control for 6 arm joints, button-controlled gripper and end-effector rotation, and contact force feedback
Integrated into device factory pattern for consistent device creation
Comprehensive unit tests with mocked WebSocket connections
Full documentation and contributor updates

Review notes:

Previous comments addressed most critical issues (race conditions, thread safety, IK joint mapping)
Implementation correctly separates IK control (6 joints) from button control (rotation + gripper)
Force feedback loop properly isolated in background thread with appropriate locking
Tests provide good coverage of device lifecycle, force limiting, and factory integration

Confidence Score: 4/5

This PR is safe to merge with minor improvements recommended for code quality
The implementation is architecturally sound with proper thread safety, comprehensive tests, and good documentation. Previous critical issues (race conditions, IK mapping) have been addressed. Score is 4/5 rather than 5/5 due to remaining style improvements noted in previous comments that would enhance maintainability but don't affect functionality
Most attention needed on source/isaaclab/isaaclab/devices/haply/se3_haply.py for the style improvements mentioned in previous comments, but no blocking issues present

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	3/5	Core Haply device implementation with WebSocket communication, thread safety issues addressed from previous comments but some style improvements needed
scripts/demos/haply_teleoperation.py	4/5	Teleoperation demo with IK control, force feedback integration, and button-based gripper/rotation control - implementation is correct
source/isaaclab/isaaclab/devices/device_base.py	5/5	Added push_force method to base class as a no-op default for haptic feedback support
source/isaaclab/test/devices/test_device_constructors.py	5/5	Comprehensive unit tests for HaplyDevice including mocked WebSocket, force limits, and factory integration

Sequence Diagram

sequenceDiagram
    participant Demo as Teleoperation Demo
    participant HaplyDev as HaplyDevice
    participant WSThread as WebSocket Thread
    participant HaplySDK as Haply SDK (WebSocket)
    participant IKCtrl as IK Controller
    participant Robot as Franka Robot
    participant Sensors as Contact Sensors

    Note over Demo,HaplySDK: Initialization Phase
    Demo->>HaplyDev: HaplyDevice(cfg)
    HaplyDev->>WSThread: Start WebSocket thread
    WSThread->>HaplySDK: Connect ws://localhost:10001
    HaplySDK-->>WSThread: Connection established
    WSThread->>HaplySDK: Subscribe to device data
    HaplySDK-->>WSThread: Device IDs (Inverse3 + VerseGrip)
    WSThread->>HaplyDev: Update cached_data (connected=True)
    HaplyDev-->>Demo: Device ready

    Note over Demo,Sensors: Teleoperation Loop (200Hz)
    loop Every simulation step
        Demo->>HaplyDev: advance()
        HaplyDev->>HaplyDev: Read cached_data (with lock)
        HaplyDev-->>Demo: [pos(3), quat(4), buttons(3)]
        
        Demo->>Demo: Extract haply_pos, button states
        Demo->>Demo: Apply coordinate mapping
        Demo->>IKCtrl: set_command(target_pos, ee_quat)
        IKCtrl->>IKCtrl: Compute IK solution
        IKCtrl-->>Demo: joint_pos_des[6]
        
        Demo->>Demo: Update joint targets (IK + rotation + gripper)
        Demo->>Robot: set_joint_position_target()
        Robot->>Robot: Execute motion
        
        Demo->>Sensors: Read contact forces
        Sensors-->>Demo: left_finger + right_finger forces
        Demo->>Demo: Average contact forces
        Demo->>HaplyDev: push_force(contact_forces)
        HaplyDev->>HaplyDev: Clip forces to limit_force (2.0 N)
        HaplyDev->>HaplyDev: Update feedback_force (with lock)
    end

    Note over WSThread,HaplySDK: Force Feedback Loop (200Hz, async)
    loop Every 1/data_rate seconds
        WSThread->>HaplySDK: Receive device state (position, orientation, buttons)
        WSThread->>HaplyDev: Update cached_data (with lock)
        WSThread->>HaplyDev: Read feedback_force (with lock)
        WSThread->>HaplySDK: Send force command to Inverse3
        HaplySDK->>HaplySDK: Apply force to haptic device
    end

    Note over Demo,HaplyDev: Cleanup
    Demo->>HaplyDev: del haply_device
    HaplyDev->>HaplyDev: Set running=False
    HaplyDev->>HaplyDev: Reset feedback_force to zero
    HaplyDev->>WSThread: Join thread (timeout 2.0s)
    WSThread->>HaplySDK: Close WebSocket
    WSThread-->>HaplyDev: Thread terminated

_{13 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

Greptile Overview

Greptile Summary

This PR adds Haply haptic device support for teleoperation with force feedback, integrating Inverse3 (position) and VerseGrip (orientation/buttons) devices via WebSocket communication.

Key Changes:

Extended DeviceBase with push_force() API supporting both tensor and dict force formats
Implemented HaplyDevice class with real-time WebSocket data streaming at 200Hz
Added comprehensive unit tests with mocked WebSocket connections
Includes teleoperation demo with IK-based robot control and force feedback

Critical Issues Found:

Race condition in advance() method (se3_haply.py:204-214) - data copied outside lock
Invalid quaternion multiplication by sensitivity (se3_haply.py:213-214) - breaks normalization
Connection logic changed from OR to AND (se3_haply.py:275) - breaks backward compatibility
Button callbacks invoked inside lock (se3_haply.py:236) - potential deadlock
Missing graceful thread shutdown and IndexError risks in device data access

Recommendations:

Address the critical race conditions and thread safety issues before merge
Fix the quaternion operation that breaks rotation data
Clarify connection requirements (AND vs OR logic for partial device operation)
Consider adding explicit cleanup method instead of relying on __del__

Confidence Score: 2/5

This PR has multiple critical concurrency and logic issues that could cause runtime errors or undefined behavior.
Score of 2/5 reflects critical issues: race conditions in data access (advance() method), invalid quaternion operations that break rotations, button callback deadlock risks, and connection logic changes that break backward compatibility. While the feature is well-tested and documented, the threading and synchronization issues pose significant runtime risks.
Pay close attention to source/isaaclab/isaaclab/devices/haply/se3_haply.py - contains multiple critical concurrency bugs and logic errors that must be resolved.

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/device_base.py	4/5	Adds `push_force()` API to DeviceBase with flexible force input formats (tensor/dict). Parameter naming could be clearer (`position` → `objects`).
source/isaaclab/isaaclab/devices/haply/se3_haply.py	2/5	New Haply device implementation with WebSocket communication and force feedback. Has critical race conditions, invalid quaternion operations, connection logic issues, and missing error handling per existing comments.
source/isaaclab/test/devices/test_device_constructors.py	5/5	Adds comprehensive unit tests for HaplyDevice with mocked WebSocket connections, testing constructor, advance(), push_force() methods, and various edge cases.

Sequence Diagram

sequenceDiagram
    participant App as Teleoperation Demo
    participant HaplyDev as HaplyDevice
    participant WS as WebSocket Thread
    participant SDK as Haply SDK
    participant Robot as Isaac Sim Robot
    
    Note over App,SDK: Initialization Phase
    App->>HaplyDev: __init__(cfg)
    HaplyDev->>WS: Start WebSocket thread
    WS->>SDK: WebSocket connect
    SDK-->>WS: Connection established
    WS->>SDK: Receive device data
    SDK-->>WS: {"inverse3": [...], "wireless_verse_grip": [...]}
    WS->>HaplyDev: Update cached_data (position, quaternion, buttons)
    HaplyDev-->>App: Device connected
    
    Note over App,SDK: Teleoperation Loop
    loop Every simulation step
        App->>HaplyDev: advance()
        HaplyDev->>HaplyDev: Lock data_lock
        HaplyDev->>HaplyDev: Read cached position, quaternion, buttons
        HaplyDev->>HaplyDev: Unlock data_lock
        HaplyDev->>HaplyDev: Execute button callbacks (outside lock)
        HaplyDev-->>App: Return tensor [pos(3), quat(4), buttons(3)]
        
        App->>App: Compute IK for robot
        App->>Robot: Set joint positions
        Robot->>Robot: Apply forces to robot
        
        App->>Robot: Read contact forces
        Robot-->>App: Force tensor
        App->>HaplyDev: push_force(forces, position=["gripper"])
        HaplyDev->>HaplyDev: Lock force_lock
        HaplyDev->>HaplyDev: Clip forces to safety limits
        HaplyDev->>HaplyDev: Update feedback_force
        HaplyDev->>HaplyDev: Unlock force_lock
        
        WS->>SDK: Send force feedback (200Hz)
        SDK-->>WS: Acknowledgment
    end
    
    Note over App,SDK: Cleanup Phase
    App->>HaplyDev: __del__()
    HaplyDev->>HaplyDev: Set running = False
    HaplyDev->>HaplyDev: Reset feedback_force to zero
    HaplyDev->>WS: Wait for thread termination
    WS->>SDK: Close WebSocket connection

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

source/isaaclab/isaaclab/devices/device_base.py

All reported issues have been resolved or clarified: - Data access properly locked in advance() method (lines 204-224) - Quaternion passed through without modification (line 210) - Button callbacks execute outside lock to prevent deadlock (lines 216-223) - Connection requires both devices by design (AND logic at line 332) - Thread cleanup handled in __del__ with proper checks (lines 144-162) No race conditions or data corruption risks present.

greptile-apps

Greptile Overview

Greptile Summary

Adds Haply haptic device support for robotic teleoperation with bidirectional force feedback, integrating Inverse3 (position tracking) and VerseGrip (orientation/buttons) devices via WebSocket communication.

Key Changes:

New HaplyDevice class with threaded WebSocket communication at 200Hz data rate
Base DeviceBase.push_force() method supporting both tensor and dict formats for haptic feedback
Complete teleoperation demo with Franka Panda arm using IK control and contact-based force feedback
Comprehensive test coverage with WebSocket mocking
Full documentation for setup and usage

Previous Issues Addressed:

Race conditions fixed by copying data inside locks (se3_haply.py:209-210)
Safe array indexing using ternary operators instead of fallback defaults (se3_haply.py:318-319)
Button callbacks moved outside data lock to prevent deadlock (se3_haply.py:215-223)
Joint mapping corrected: 6 joints from IK + separate joint7 rotation control (haply_teleoperation.py:313-314)
Thread cleanup improved with explicit join and timeout (se3_haply.py:157-161)

Confidence Score: 4/5

Safe to merge with minor threading considerations
The implementation is well-designed with proper thread synchronization, comprehensive error handling, and thorough testing. Most critical race conditions and logic errors from previous review have been resolved. The code demonstrates good practices with lock management, safe data copying, and proper cleanup. Minor deduction for the daemon thread fallback in __del__ which could leave threads running in edge cases, but this is a reasonable safety measure.
No files require special attention - the implementation is solid across all components

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	4/5	Core Haply device implementation with WebSocket communication, threading, and force feedback - most previous critical issues addressed
scripts/demos/haply_teleoperation.py	5/5	Teleoperation demo with IK control and force feedback - clean implementation with proper joint mapping
source/isaaclab/isaaclab/devices/device_base.py	5/5	Added `push_force` base method with dual tensor/dict format support for haptic devices
source/isaaclab/test/devices/test_device_constructors.py	5/5	Test coverage added for Haply device with proper WebSocket mocking

Sequence Diagram

sequenceDiagram
    participant User
    participant Demo as haply_teleoperation.py
    participant Haply as HaplyDevice
    participant WS as WebSocket Thread
    participant SDK as Haply SDK
    participant IK as IK Controller
    participant Robot as Franka Robot
    participant Sensor as Contact Sensors

    User->>Demo: Start demo script
    Demo->>Haply: Initialize HaplyDevice(cfg)
    Haply->>WS: Start WebSocket thread
    WS->>SDK: Connect to ws://localhost:10001
    SDK-->>WS: Connection established
    WS->>SDK: Request device data
    SDK-->>WS: Inverse3 + VerseGrip data
    WS->>Haply: Update cached_data (position, quaternion, buttons)
    Haply-->>Demo: Device ready

    loop Simulation Loop (200Hz)
        Demo->>Haply: advance()
        Haply->>Haply: Lock data_lock
        Haply->>Haply: Copy position, quaternion, buttons
        Haply->>Haply: Unlock data_lock
        Haply->>Haply: Execute button callbacks
        Haply-->>Demo: Return [pos(3), quat(4), buttons(3)]
        
        Demo->>Demo: Process buttons (gripper, rotation)
        Demo->>Demo: Apply coordinate mapping
        Demo->>IK: compute(target_pos, ee_quat, jacobian)
        IK-->>Demo: Joint positions (6 joints)
        Demo->>Robot: set_joint_position_target(joints + ee_rotation + gripper)
        
        Robot->>Sensor: Update contact forces
        Sensor-->>Demo: Get net_forces_w
        Demo->>Demo: Compute total_contact_force
        Demo->>Haply: push_force(total_contact_force)
        Haply->>Haply: Lock force_lock
        Haply->>Haply: Clip and store feedback_force
        Haply->>Haply: Unlock force_lock
        
        WS->>Haply: Copy feedback_force
        WS->>SDK: Send set_cursor_force command
        SDK->>SDK: Apply force to Inverse3
    end

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

mingxueg-nv · 2025-11-04T10:23:43Z

@rwiltz Yes I think that could work, you could also add the torque into the signature as well if you'd like: ))

Hi, @rwiltz @ooctipus , the specific location for force feedback also depends on where we've placed the contact sensors. In the demo, I added them to left and right fingers on the end-effector.
So I've incorporated your suggestion by adding a position parameter to specify which sensor's force to feedback. (In the demo script, I'm currently using the default end-effector for now.)

source/isaaclab/isaaclab/devices/device_base.py

greptile-apps

Greptile Overview

Greptile Summary

This PR refactors the push_force API to simplify force feedback handling by replacing the dict-based interface with direct tensor indexing.

Key Changes:

Removed dict format support from push_force in both DeviceBase and HaplyDevice
Changed position parameter from list[str] to torch.Tensor for index-based force selection
Updated documentation to reflect tensor-only interface
Modified test cases to use tensor indexing with 1D index tensors

Improvements:

Simpler, more consistent API using native PyTorch operations
Better performance by eliminating dict lookups and conversions
More explicit force aggregation using .sum(dim=0) on selected forces

Issue Found:

Potential edge case with 0-dimensional position tensors that could cause dimension mismatch in sum(dim=0) operation

Confidence Score: 4/5

Safe to merge with minor fix recommended for edge case handling
The refactoring successfully simplifies the API and all existing threading/locking issues were addressed in previous commits. However, there's a logical edge case with 0-D position tensors that should be handled before merging to prevent potential runtime errors
Pay attention to source/isaaclab/isaaclab/devices/haply/se3_haply.py line 264 for the tensor indexing edge case

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/device_base.py	5/5	Added `push_force` API method to base class with simplified tensor-based interface for haptic feedback
source/isaaclab/isaaclab/devices/haply/se3_haply.py	3/5	Refactored `push_force` to use tensor indexing instead of dict format; potential issue with 0-D position tensors
source/isaaclab/test/devices/test_device_constructors.py	5/5	Updated test cases to use tensor indexing for force selection; comprehensive coverage of single and multi-index cases

Sequence Diagram

sequenceDiagram
    participant App as Application/Demo
    participant Haply as HaplyDevice
    participant WebSocket as WebSocket Thread
    participant SDK as Haply SDK
    participant Hardware as Haply Hardware

    Note over App,Hardware: Initialization Phase
    App->>Haply: __init__(cfg)
    Haply->>WebSocket: _start_websocket_thread()
    WebSocket->>SDK: Connect (ws://localhost:10001)
    SDK-->>WebSocket: Connection established
    WebSocket->>SDK: Receive device data
    SDK-->>WebSocket: Device IDs + State
    WebSocket->>Haply: Update cached_data (with data_lock)
    Haply-->>App: Ready (both devices connected)

    Note over App,Hardware: Teleoperation Loop
    loop Every frame
        App->>Haply: advance()
        Haply->>Haply: Lock data_lock
        Haply->>Haply: Copy position, quaternion, buttons
        Haply->>Haply: Release data_lock
        Haply->>Haply: Execute button callbacks (outside lock)
        Haply-->>App: Return tensor [pos, quat, buttons]
        
        App->>App: Compute robot control
        App->>App: Read contact forces from simulation
        App->>Haply: push_force(forces, position)
        Haply->>Haply: Select forces by indices
        Haply->>Haply: Sum selected forces
        Haply->>Haply: Clip to [-limit_force, limit_force]
        Haply->>Haply: Lock force_lock
        Haply->>Haply: Update feedback_force
        Haply->>Haply: Release force_lock
    end

    Note over WebSocket,Hardware: Background Force Feedback
    loop Every 1/data_rate seconds
        WebSocket->>Haply: Lock force_lock
        WebSocket->>Haply: Copy feedback_force
        WebSocket->>Haply: Release force_lock
        WebSocket->>SDK: Send force command
        SDK->>Hardware: Apply force to Inverse3
        Hardware-->>SDK: Acknowledgment
    end

    Note over App,Hardware: Cleanup
    App->>Haply: __del__()
    Haply->>Haply: Set running = False
    Haply->>Haply: Reset feedback_force to zero
    Haply->>WebSocket: Wait for thread termination (2s timeout)
    WebSocket->>SDK: Close connection
    SDK->>Hardware: Stop force feedback

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

source/isaaclab/isaaclab/devices/haply/se3_haply.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: mingxueg <mingxueg@nvidia.com>

greptile-apps

Greptile Overview

Greptile Summary

Adds comprehensive Haply haptic device support for teleoperation with force feedback. The implementation provides a thread-safe WebSocket-based interface to Haply Inverse3 (position tracking + force feedback) and VerseGrip (orientation + button inputs) devices.

Key implementation details:

WebSocket communication loop running at 200Hz for real-time haptic feedback
Thread-safe data handling with separate locks for position/orientation data, force feedback, and connection state
Proper error handling for device disconnection and WebSocket timeouts (marks disconnected after 10 consecutive 1s timeouts)
Force feedback safety limits (configurable, default 2.0N) with clipping
Button callback system that executes outside critical locks to prevent deadlock
Graceful cleanup in __del__ with thread joining and force reset

Critical fixes applied in recent commits:

Fixed race condition by copying data within lock (se3_haply.py:208-213)
Fixed IndexError risk with safe ternary operators for empty device lists (se3_haply.py:307-308)
Fixed callback deadlock by executing callbacks outside data_lock (se3_haply.py:215-223)
Fixed scalar tensor handling in push_force with proper unsqueeze (se3_haply.py:264)
Removed invalid quaternion scaling that broke normalization
Properly wrapped websockets import in try-except

Minor style issue:

Inconsistent dictionary indentation formatting (se3_haply.py:358-363)

Confidence Score: 4/5

This PR is safe to merge with low risk - all critical threading and safety issues from previous reviews have been properly addressed
Score reflects thorough resolution of critical issues (race conditions, deadlocks, IndexError risks, quaternion normalization) through multiple review cycles. The implementation demonstrates solid thread-safety with appropriate lock usage, proper error handling for device disconnection, and safety limits for force feedback. Only minor style issues remain (formatting inconsistency). The code has undergone extensive iteration based on feedback and is production-ready for haptic device teleoperation.
No files require special attention - all critical issues have been resolved

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	4/5	Added complete Haply device API with WebSocket communication, force feedback, thread-safe data handling, and proper error handling. Most critical race conditions and safety issues from previous reviews have been resolved.

Sequence Diagram

sequenceDiagram
    participant App as Teleoperation App
    participant HD as HaplyDevice
    participant WST as WebSocket Thread
    participant SDK as Haply SDK
    participant I3 as Inverse3 Device
    participant VG as VerseGrip Device
    
    App->>HD: __init__(cfg)
    HD->>HD: Initialize locks, cached_data
    HD->>WST: Start WebSocket thread (daemon=False)
    
    loop WebSocket Loop (200Hz)
        WST->>SDK: Connect via WebSocket
        SDK-->>WST: Connection established
        
        WST->>SDK: recv() with 1s timeout
        SDK-->>WST: JSON data {inverse3: [...], wireless_verse_grip: [...]}
        
        WST->>WST: Parse position from Inverse3
        WST->>WST: Parse quaternion & buttons from VerseGrip
        
        WST->>HD: Update cached_data (within data_lock)
        WST->>HD: Update connected flag (inverse3_connected AND versegrip_connected)
        
        WST->>HD: Read feedback_force (within force_lock)
        WST->>SDK: send({"inverse3": [{"commands": {"set_cursor_force": ...}}]})
        SDK->>I3: Apply force feedback
        
        WST->>WST: sleep(1/data_rate)
    end
    
    App->>HD: advance()
    HD->>HD: Check both devices connected
    HD->>HD: Copy position, quaternion, buttons (within data_lock)
    HD->>HD: Execute button callbacks (OUTSIDE lock)
    HD-->>App: torch.Tensor [x,y,z,qx,qy,qz,qw,btn_a,btn_b,btn_c]
    
    App->>HD: push_force(forces, position)
    HD->>HD: Select and sum forces at position indices
    HD->>HD: Clip to [-limit_force, +limit_force]
    HD->>HD: Update feedback_force (within force_lock)
    
    App->>HD: __del__()
    HD->>HD: Set running = False
    HD->>HD: Reset feedback_force to zero
    HD->>WST: Wait for thread.join(timeout=2s)
    WST->>SDK: Close WebSocket connection

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

source/isaaclab/isaaclab/devices/haply/se3_haply.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: mingxueg <mingxueg@nvidia.com>

greptile-apps

Greptile Overview

Greptile Summary

Adds Haply haptic device API for robotic teleoperation with force feedback. The implementation provides WebSocket-based communication with Haply Inverse3 (position tracking and force feedback) and VerseGrip (orientation and button inputs) devices.

Key changes:

New HaplyDevice class extending DeviceBase with SE(3) control and force feedback
Real-time bidirectional communication at 200Hz via WebSocket (asyncio + threading)
Thread-safe data exchange using multiple locks (data_lock, force_lock, _connected_lock)
Button callback system for teleoperation control
Safety limits for force feedback (configurable limit_force parameter)

Issues identified:

Threading concerns: daemon=False thread may become zombie if join timeout expires
Incomplete comment at line 348 explaining AND logic requirement
Multiple race conditions and concurrency issues flagged in previous comments need attention

Confidence Score: 2/5

This PR introduces multiple critical threading and concurrency issues that could cause race conditions, deadlocks, or undefined behavior in production
Score reflects several critical logic issues flagged in previous comments including race conditions (line 204-214, 331), threading problems (daemon flag inconsistency), potential deadlocks (callbacks inside locks at line 236), IndexError risks (line 288), and invalid quaternion operations. While the feature adds valuable functionality, these concurrency and safety issues need resolution before merge
source/isaaclab/isaaclab/devices/haply/se3_haply.py requires careful attention to threading/locking patterns and race condition fixes

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	3/5	New Haply device integration with WebSocket communication, force feedback, and teleoperation. Several threading/concurrency issues and incomplete comments remain.

Sequence Diagram

sequenceDiagram
    participant App as Teleoperation Demo
    participant HD as HaplyDevice
    participant WT as WebSocket Thread
    participant SDK as Haply SDK
    participant I3 as Inverse3 Device
    participant VG as VerseGrip Device

    App->>HD: __init__(cfg)
    HD->>HD: Initialize locks & data structures
    HD->>WT: Start WebSocket thread (daemon=False)
    WT->>SDK: Connect WebSocket (ws://localhost:10001)
    
    loop Every 5ms (200Hz)
        SDK->>WT: Send device state (JSON)
        WT->>WT: Parse Inverse3 position
        WT->>WT: Parse VerseGrip orientation & buttons
        WT->>HD: Update cached_data (with data_lock)
        WT->>SDK: Send force feedback command
    end

    loop Simulation Loop
        App->>HD: advance()
        HD->>HD: Read cached_data (with data_lock)
        HD->>HD: Execute button callbacks (outside lock)
        HD-->>App: Return [pos(3), quat(4), buttons(3)]
        
        App->>App: Compute IK & force feedback
        App->>HD: push_force(forces, position)
        HD->>HD: Clip forces to limit_force
        HD->>HD: Update feedback_force (with force_lock)
    end

    App->>HD: __del__()
    HD->>HD: Set running=False
    HD->>HD: Reset feedback_force to zero
    HD->>WT: Wait for thread.join(timeout=2.0)
    WT->>SDK: Close WebSocket connection

_{1 file reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

source/isaaclab/isaaclab/devices/haply/se3_haply.py

Current implementation addresses all concurrency concerns: - Locked data access with safe copying patterns - Callbacks execute outside locks (no deadlock risk) - Proper boundary checks prevent IndexErrors - Quaternion data preserved without modification - Thread cleanup with defensive checks All threading and safety patterns verified for production use.

greptile-apps

Greptile Overview

Greptile Summary

This PR adds comprehensive Haply haptic device support for teleoperation with force feedback. The implementation introduces:

Core Device API (se3_haply.py): WebSocket-based communication with Inverse3 (position/force) and VerseGrip (orientation/buttons) devices, including thread-safe data handling and real-time force feedback at 200Hz
Teleoperation Demo (haply_teleoperation.py): Franka Panda arm control using IK for 6-DOF positioning, button-based gripper and wrist rotation control, and contact force feedback
Base Device Extension (device_base.py): Added push_force() method to device base class for haptic feedback support

The implementation has been refined through multiple iterations addressing threading concerns, race conditions, and synchronization issues. Most previous comments have been addressed with fixes for data locking, connection logic (AND requirement for both devices), and array indexing safety.

Key Technical Aspects:

Threading: WebSocket runs in separate thread with proper lock management for cached_data, feedback_force, and connection status
Force Feedback: Contact forces from gripper fingers are averaged and pushed to Inverse3 with 2N safety clipping
IK Control: 6 arm joints controlled via differential IK, wrist rotation (joint 7) via button C, gripper via buttons A/B
Coordinate Mapping: Haply workspace transformed to robot workspace with safety limits

Confidence Score: 4/5

This PR is safe to merge with minor review of previous comments
The implementation is well-structured with proper threading, lock management, and safety limits. Most critical issues identified in previous comments (race conditions, connection logic, array indexing) have been addressed in recent commits. The code includes comprehensive error handling, device connection monitoring, and force feedback clipping for safety. The remaining previous comments are primarily style suggestions and minor improvements rather than blocking issues.
Review previous comments on source/isaaclab/isaaclab/devices/haply/se3_haply.py - most have been addressed but should be verified for completeness

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	4/5	Core Haply device implementation with WebSocket communication, force feedback, and threading - previous comments addressed most issues
scripts/demos/haply_teleoperation.py	4/5	Teleoperation demo with IK control and force feedback - joint indexing and button handling work correctly
source/isaaclab/isaaclab/devices/device_base.py	5/5	Added push_force base method with clear documentation - implementation is clean and extensible

Sequence Diagram

sequenceDiagram
    participant User
    participant Demo as haply_teleoperation.py
    participant Device as HaplyDevice
    participant WSThread as WebSocket Thread
    participant SDK as Haply SDK
    participant Sim as Isaac Sim
    participant Robot as Franka Robot
    participant Sensors as Contact Sensors

    Note over Demo,SDK: Initialization Phase
    Demo->>Device: HaplyDevice(cfg)
    Device->>WSThread: Start websocket_thread (daemon=False)
    WSThread->>SDK: websockets.connect(uri)
    SDK-->>WSThread: Connection established
    WSThread->>SDK: recv() - get device IDs
    WSThread->>Device: Update cached_data with device IDs
    WSThread->>Device: Set connected=True (both devices)

    Note over Demo,Sensors: Teleoperation Loop (200Hz)
    loop Every simulation step
        Demo->>Device: advance()
        Device->>Device: Lock data_lock
        Device->>Device: Copy position, quaternion, buttons
        Device->>Device: Unlock data_lock
        Device->>Device: Execute button callbacks (outside lock)
        Device-->>Demo: Return [pos(3), quat(4), buttons(3)]
        
        Demo->>Demo: Extract button states
        Demo->>Demo: Update gripper_target & ee_rotation_angle
        Demo->>Demo: apply_haply_to_robot_mapping()
        Demo->>Demo: IK compute for 6 arm joints
        Demo->>Robot: set_joint_position_target (joints 0-6 + gripper)
        
        Demo->>Sim: sim.step() x5
        Sim->>Robot: Update physics
        Sim->>Sensors: Update contact forces
        
        Demo->>Sensors: Get net_forces_w (left & right fingers)
        Demo->>Demo: Average forces: (left + right) * 0.5
        Demo->>Device: push_force(total_contact_force)
        Device->>Device: Lock force_lock
        Device->>Device: Clip force to [-2N, 2N]
        Device->>Device: Update feedback_force dict
        Device->>Device: Unlock force_lock
    end

    Note over WSThread,SDK: Force Feedback Loop (200Hz)
    loop WebSocket loop
        WSThread->>SDK: recv(timeout=1s)
        SDK-->>WSThread: Device state data (JSON)
        WSThread->>Device: Lock data_lock
        WSThread->>Device: Update cached_data (position, quat, buttons)
        WSThread->>Device: Unlock data_lock
        
        WSThread->>Device: Lock force_lock
        WSThread->>Device: Copy feedback_force
        WSThread->>Device: Unlock force_lock
        WSThread->>SDK: send(set_cursor_force command)
        WSThread->>WSThread: asyncio.sleep(1/data_rate)
    end

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps

Greptile Overview

Greptile Summary

This PR adds comprehensive Haply haptic device support for teleoperation with force feedback in IsaacLab. The implementation provides a HaplyDevice class that interfaces with both Inverse3 (for position tracking and force feedback) and VerseGrip (for orientation and button inputs) via WebSocket communication.

Key changes:

New HaplyDevice class with WebSocket-based async communication running at 200Hz
Thread-safe data exchange using locks for position, orientation, button states, and force feedback
Force feedback with configurable safety limits (default 2.0N)
Button callback system for teleoperation controls
Connection monitoring with timeout detection and auto-reconnection
Demo script implementing IK-based teleoperation with force feedback

Critical issues identified in previous comments:

Multiple race conditions in data access between WebSocket thread and main thread
Thread safety issues with button callbacks potentially causing deadlocks
Invalid quaternion scaling that breaks normalization
Index error risks with empty device arrays
Thread cleanup issues in __del__ method

The implementation demonstrates good understanding of WebSocket communication patterns and thread synchronization, but requires fixes to the race conditions and thread safety issues before production use.

Confidence Score: 3/5

This PR is moderately safe to merge but requires fixes to critical race conditions and thread safety issues
Score reflects a well-structured implementation with good documentation and comprehensive testing, but multiple critical thread safety issues have been identified that could cause runtime errors or undefined behavior. The race conditions in data access (line 204-214), button callback deadlock risks (line 216-223), and invalid quaternion scaling are serious issues that need resolution. However, the feature is isolated to a new module with no impact on existing functionality, and the issues are well-documented in previous comments with clear fixes available.
source/isaaclab/isaaclab/devices/haply/se3_haply.py requires attention for race condition fixes, particularly around data_lock usage in advance() method and button callback execution

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	3/5	New Haply device API with WebSocket communication, force feedback, and button callbacks. Multiple race conditions and thread safety issues already flagged in previous comments.

Sequence Diagram

sequenceDiagram
    participant User
    participant HaplyDevice
    participant WebSocketThread
    participant HaplySDK
    participant Inverse3
    participant VerseGrip

    User->>HaplyDevice: __init__(cfg)
    HaplyDevice->>WebSocketThread: Start thread (daemon=False)
    WebSocketThread->>HaplySDK: Connect WebSocket
    
    loop Every 1/data_rate (200Hz)
        HaplySDK->>WebSocketThread: Send device state
        WebSocketThread->>WebSocketThread: Parse JSON data
        WebSocketThread->>WebSocketThread: Lock data_lock
        WebSocketThread->>HaplyDevice: Update cached_data
        WebSocketThread->>WebSocketThread: Unlock data_lock
        
        WebSocketThread->>WebSocketThread: Lock force_lock
        WebSocketThread->>WebSocketThread: Copy feedback_force
        WebSocketThread->>WebSocketThread: Unlock force_lock
        WebSocketThread->>HaplySDK: Send force command
    end
    
    User->>HaplyDevice: advance()
    HaplyDevice->>HaplyDevice: Lock data_lock
    HaplyDevice->>HaplyDevice: Copy position, quaternion, buttons
    HaplyDevice->>HaplyDevice: Unlock data_lock
    HaplyDevice->>HaplyDevice: Execute button callbacks
    HaplyDevice->>User: Return tensor[pos, quat, buttons]
    
    User->>HaplyDevice: push_force(forces)
    HaplyDevice->>HaplyDevice: Clip forces to limit
    HaplyDevice->>HaplyDevice: Lock force_lock
    HaplyDevice->>HaplyDevice: Update feedback_force
    HaplyDevice->>HaplyDevice: Unlock force_lock
    
    User->>HaplyDevice: __del__()
    HaplyDevice->>HaplyDevice: Set running=False
    HaplyDevice->>HaplyDevice: Reset feedback_force
    HaplyDevice->>WebSocketThread: join(timeout=2.0)
    alt Thread still alive
        HaplyDevice->>WebSocketThread: Set daemon=True
    end

_{1 file reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

Current implementation addresses all concurrency concerns: - Locked data access with safe copying patterns - Callbacks execute outside locks (no deadlock risk) - Proper boundary checks prevent IndexErrors - Quaternion data preserved without modification - Thread cleanup with defensive checks All threading and safety patterns verified for production use.

greptile-apps

Greptile Overview

Greptile Summary

This PR adds comprehensive Haply haptic device support for robotic teleoperation with force feedback in Isaac Lab. The implementation includes a HaplyDevice class that communicates with Haply Inverse3 and VerseGrip devices via WebSocket, providing position tracking, orientation sensing, button inputs, and bidirectional force feedback.

Key changes:

New HaplyDevice class with WebSocket-based communication running at 200Hz
Teleoperation demo script integrating Haply with Franka Panda robot using IK control
Extension to DeviceBase.push_force() interface supporting force feedback devices
Comprehensive test suite with mocked WebSocket, threading, and asyncio components
Documentation and usage examples

Architecture: The implementation uses a background thread for WebSocket communication, with proper locking mechanisms (data_lock, force_lock, _connected_lock) to synchronize data between the WebSocket thread and the main simulation loop. The demo script maps Haply workspace coordinates to robot end-effector positions via IK, with button-controlled gripper and end-effector rotation, and sends contact forces back to the Haply device for haptic feedback.

Previous review rounds have identified several threading and concurrency concerns that should be addressed before merging, particularly around race conditions in data access and thread lifecycle management.

Confidence Score: 3/5

This PR has architectural merit but contains several critical threading and concurrency issues that must be resolved before safe deployment
Score of 3/5 reflects that while the implementation is well-documented with comprehensive tests, previous review rounds have identified multiple critical issues: race conditions in advance() where data is accessed outside locks, potential deadlocks from callbacks executed inside locks, AND logic change breaking backward compatibility, incomplete thread cleanup in __del__, and missing validation for 0-D position tensors. The core architecture is sound and the feature adds valuable functionality, but the threading concerns pose real risks in production use.
Primary attention needed: source/isaaclab/isaaclab/devices/haply/se3_haply.py for resolving threading/concurrency issues identified in previous comments

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	3/5	Core Haply device implementation with WebSocket communication, threading, and force feedback. Previous comments identified critical race conditions, thread safety issues, and connection handling concerns that need resolution.
scripts/demos/haply_teleoperation.py	4/5	Teleoperation demo script for Haply device with IK control. Joint indexing logic correctly handles 6 IK joints + 1 button-controlled joint. Minor style improvements suggested in previous comments.
source/isaaclab/isaaclab/devices/device_base.py	5/5	Base class for teleoperation devices with abstract interface. Minor parameter naming issue noted in previous comments (`position` parameter could be renamed to `objects` or `keys` for clarity).
source/isaaclab/test/devices/test_device_constructors.py	5/5	Comprehensive test suite for Haply device constructors with mocking for WebSocket, threading, and asyncio. Tests cover config validation, force limiting, and position tensor handling.

Sequence Diagram

sequenceDiagram
    participant User
    participant Demo as haply_teleoperation.py
    participant HaplyDevice
    participant WSThread as WebSocket Thread
    participant HaplySDK as Haply SDK (WebSocket)
    participant Robot as Franka Robot
    participant IK as IK Controller
    participant Sensors as Contact Sensors

    User->>Demo: Start teleoperation demo
    Demo->>HaplyDevice: __init__(cfg)
    HaplyDevice->>WSThread: Start WebSocket thread (daemon=False)
    WSThread->>HaplySDK: Connect via WebSocket
    HaplySDK-->>WSThread: Connection established
    WSThread->>HaplySDK: Receive device data (200Hz)
    HaplySDK-->>WSThread: {position, orientation, buttons}
    WSThread->>HaplyDevice: Update cached_data (with data_lock)
    HaplyDevice-->>Demo: Device ready

    loop Simulation Loop (200Hz)
        Demo->>HaplyDevice: advance()
        HaplyDevice->>HaplyDevice: Copy cached_data (with data_lock)
        HaplyDevice->>HaplyDevice: Execute button callbacks
        HaplyDevice-->>Demo: [pos, quat, buttons] tensor
        
        Demo->>Demo: Process button states (gripper, rotation)
        Demo->>Demo: apply_haply_to_robot_mapping()
        Demo->>IK: set_command(target_pos, ee_quat)
        Demo->>IK: compute(ee_pos, ee_quat, jacobian, joint_pos)
        IK-->>Demo: joint_pos_des (6 joints)
        
        Demo->>Robot: set_joint_position_target(joints 0-5 from IK, joint 6 from button, gripper)
        Robot->>Robot: Execute motion (5 sim steps)
        
        Robot->>Sensors: Update contact forces
        Sensors-->>Demo: net_forces_w (left + right finger)
        Demo->>Demo: Average contact forces
        Demo->>HaplyDevice: push_force(total_contact_force)
        HaplyDevice->>HaplyDevice: Update feedback_force (with force_lock)
        
        WSThread->>HaplyDevice: Copy feedback_force (with force_lock)
        WSThread->>HaplySDK: Send force command {"set_cursor_force": values}
        HaplySDK->>HaplySDK: Apply force to Inverse3
    end

    User->>Demo: Stop simulation
    Demo->>HaplyDevice: __del__()
    HaplyDevice->>HaplyDevice: Set running=False
    HaplyDevice->>WSThread: Wait for thread.join(timeout=2.0)
    WSThread->>HaplySDK: Close WebSocket connection
    WSThread-->>HaplyDevice: Thread terminated

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

All reported issues have been resolved or clarified: - Data access properly locked in advance() method (lines 204-224) - Quaternion passed through without modification (line 210) - Button callbacks execute outside lock to prevent deadlock (lines 216-223) - Connection requires both devices by design (AND logic at line 332) - Thread cleanup handled in __del__ with proper checks (lines 144-162) No race conditions or data corruption risks present.

greptile-apps

Greptile Overview

Greptile Summary

This PR adds comprehensive support for Haply haptic devices (Inverse3 + VerseGrip) for robotic teleoperation with bidirectional force feedback in Isaac Lab.

Key Changes:

New HaplyDevice class with WebSocket-based communication at 200Hz data rate
Thread-safe data handling with proper locking for position, orientation, button states, and force feedback
Demo script showcasing Franka Panda teleoperation with IK control, contact-based force feedback, and intuitive button mappings
Comprehensive test coverage with mocked WebSocket connections
Complete documentation including usage guide and API reference

Implementation Highlights:

Resolved previous race conditions by copying data within lock critical sections (se3_haply.py:204-213)
Fixed connection logic to require both Inverse3 and VerseGrip devices (AND logic) for full functionality
Proper cleanup in __del__ with graceful thread shutdown and force reset
IK controller manages 6 arm joints (panda_joint1-6) while Button C independently controls end-effector rotation (panda_joint7)

Safety Features:

Force feedback clamped to configurable limits (default 2.0N)
Workspace limits enforced for robot safety
Connection timeout monitoring with automatic disconnection after 10 consecutive timeouts

Confidence Score: 4/5

This PR is safe to merge with minor considerations for production deployments
The implementation demonstrates solid engineering with proper thread safety, comprehensive testing, and good documentation. Previous critical issues (race conditions, connection logic, data copying) have been addressed. The score reflects that while the core functionality is sound, the WebSocket thread cleanup could be more robust (daemon flag fallback after timeout), and the AND logic for device connection is intentional but breaks backward compatibility if users expect partial operation with only one device.
Pay attention to se3_haply.py:285 (daemon=False thread) and se3_haply.py:159-161 (thread cleanup fallback) if deployment stability issues arise

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/devices/haply/se3_haply.py	4/5	Core Haply device implementation with WebSocket communication, thread-safe data handling, and force feedback. Previous race condition and connection logic issues have been resolved with proper locking and data copying within critical sections.
scripts/demos/haply_teleoperation.py	5/5	Teleoperation demo script correctly implementing IK control with 6 joints, gripper control, and force feedback. Joint indexing is correct by design - IK controls panda_joint1-6 while button C controls panda_joint7 rotation.
source/isaaclab/isaaclab/devices/device_base.py	5/5	Base class for teleoperation devices with proper abstraction for force feedback support. Clean interface design with optional retargeting functionality.
source/isaaclab/test/devices/test_device_constructors.py	5/5	Comprehensive test suite with proper mocking for all device types including Haply. Tests cover configuration, connection scenarios, force feedback, and edge cases.

Sequence Diagram

sequenceDiagram
    participant User
    participant Demo as haply_teleoperation.py
    participant HaplyDevice as HaplyDevice
    participant WebSocketThread as WebSocket Thread
    participant HaplySDK as Haply SDK (WS Server)
    participant Robot as Franka Robot
    participant Sensors as Contact Sensors

    User->>Demo: Launch demo script
    Demo->>HaplyDevice: __init__(cfg)
    HaplyDevice->>WebSocketThread: _start_websocket_thread()
    activate WebSocketThread
    WebSocketThread->>HaplySDK: Connect via WebSocket
    
    loop Wait for connection (5s timeout)
        WebSocketThread->>HaplySDK: Receive device data
        HaplySDK-->>WebSocketThread: {inverse3: {...}, wireless_verse_grip: {...}}
        WebSocketThread->>WebSocketThread: Update cached_data (with data_lock)
        WebSocketThread->>WebSocketThread: Set connected = True (both devices)
    end
    
    HaplyDevice-->>Demo: Device ready
    
    loop Teleoperation Loop (200Hz sim)
        Demo->>HaplyDevice: advance()
        HaplyDevice->>HaplyDevice: Read cached_data (with data_lock)
        HaplyDevice-->>Demo: [pos, quat, buttons]
        
        Demo->>Demo: Map Haply pos to robot workspace
        Demo->>Demo: IK compute (6 joints)
        Demo->>Demo: Button C controls joint7 rotation
        Demo->>Robot: set_joint_position_target()
        
        Demo->>Sensors: Get contact forces
        Sensors-->>Demo: left_finger + right_finger forces
        Demo->>HaplyDevice: push_force(total_contact_force)
        HaplyDevice->>HaplyDevice: Update feedback_force (with force_lock)
        
        WebSocketThread->>WebSocketThread: Read feedback_force (with force_lock)
        WebSocketThread->>HaplySDK: Send force command
        HaplySDK-->>User: Apply haptic feedback
    end
    
    Demo->>HaplyDevice: __del__()
    HaplyDevice->>HaplyDevice: Reset feedback_force to zero
    HaplyDevice->>WebSocketThread: Set running = False
    deactivate WebSocketThread

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

mingxueg-nv added 5 commits October 29, 2025 12:23

draft demo with haply teleop with contact sensor

f73c475

Update Ik control

4220be1

enhance scene config and log

e3beabd

Merge branch 'isaac-sim:main' into mingxueg/add_haply_devices_for_teleop

043878d

fix format

af0ee44

mingxueg-nv requested a review from ooctipus as a code owner October 29, 2025 07:49

github-actions bot added isaac-mimic Related to Isaac Mimic team isaac-lab Related to Isaac Lab team labels Oct 29, 2025

greptile-apps bot reviewed Oct 29, 2025

View reviewed changes

Remove unnecessary log

128c792

greptile-apps bot reviewed Oct 29, 2025

View reviewed changes

source/isaaclab/isaaclab/devices/haply/se3_haply.py Show resolved Hide resolved

source/isaaclab/isaaclab/devices/haply/se3_haply.py Show resolved Hide resolved

resolve comments from greptile

d914e24

greptile-apps bot reviewed Oct 29, 2025

View reviewed changes

source/isaaclab/isaaclab/devices/haply/se3_haply.py Outdated Show resolved Hide resolved

source/isaaclab/isaaclab/devices/haply/se3_haply.py Outdated Show resolved Hide resolved

Update timeout check method

8a46ce3

greptile-apps bot reviewed Oct 29, 2025

View reviewed changes

improve thread safety and cleanup in Haply device

8a5a2f8

greptile-apps bot reviewed Oct 29, 2025

View reviewed changes

source/isaaclab/isaaclab/devices/haply/se3_haply.py Outdated Show resolved Hide resolved

Use more stable method to check connect

635373e

greptile-apps bot reviewed Oct 29, 2025

View reviewed changes

source/isaaclab/isaaclab/devices/haply/se3_haply.py Outdated Show resolved Hide resolved

source/isaaclab/isaaclab/devices/haply/se3_haply.py Outdated Show resolved Hide resolved

Fix thread safety and device disconnection detection

4a50a6f

greptile-apps bot reviewed Oct 29, 2025

View reviewed changes

Merge branch 'main' into mingxueg/add_haply_devices_for_teleop

d85c938

greptile-apps bot reviewed Oct 30, 2025

View reviewed changes

source/isaaclab/isaaclab/devices/haply/se3_haply.py Show resolved Hide resolved

Update source/isaaclab/isaaclab/devices/haply/se3_haply.py

243b8ae

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: mingxueg-nv <mingxueg@nvidia.com>

greptile-apps bot reviewed Oct 30, 2025

View reviewed changes

Add test script for haply device

544e41a

greptile-apps bot reviewed Oct 30, 2025

View reviewed changes

scripts/demos/haply_teleoperation.py Outdated Show resolved Hide resolved

fixed ik joints index issue

13c4047

greptile-apps bot reviewed Oct 30, 2025

View reviewed changes

scripts/demos/haply_teleoperation.py Outdated Show resolved Hide resolved

scripts/demos/haply_teleoperation.py Outdated Show resolved Hide resolved

rwiltz reviewed Oct 30, 2025

View reviewed changes

source/isaaclab/isaaclab/devices/haply/se3_haply.py Outdated Show resolved Hide resolved

Merge branch 'main' into mingxueg/add_haply_devices_for_teleop

d1dd6bb