[CANDLE] Did not receive response from CANdle! [FAILED]

[missxa]

When powering on the Jetson Orin and auto starting a script using pyCANdle, an error is reported: Did not receive response from CANdle! [FAILED]. When this happens, the drives do not respond to position commands.

1. How to detect this failure?
2. What is the cause of it and how can it be avoided?

Logs:

[CANDLE] CANdle library version: v3.4.1
[CANDLE] Device firmware version: v2.2.1
[CANDLE] CANdle at /dev/ttyACM0, ID: 0xac12a51e374c1f30 ready (USB)
[CANDLE] Starting pinging drives at baudrate: 1M
[CANDLE] Found drives.
[CANDLE] 1: ID = 144 (0x90)
[CANDLE] 2: ID = 145 (0x91)
[CANDLE] Added MD80 with ID: 145  [OK]
[CANDLE] Added MD80 with ID: 144  [OK]
[CANDLE] Setting new zero position successful at ID: 145  [OK]
[CANDLE] Setting control mode successful at ID: 145  [OK]
[CANDLE] Enabling successful at ID: 145  [OK]
[CANDLE] Setting new zero position successful at ID: 144  [OK]
[CANDLE] Setting control mode successful at ID: 144  [OK]
[CANDLE] Enabling successful at ID: 144  [OK]
[CANDLE] Beginnig auto update loop mode  [OK]
[CANDLE] Did not receive response from CANdle!  [FAILED]

I'm using the following code to detect if there are any errors with the drives, but motors don't seem to report any errors when Did not receive response from CANdle! [FAILED] happens

quick_status = self.candle.md80s[id].getQuickStatus()
error_bits = {
    'main_encoder_error': (quick_status & (1 << 0)) != 0,
    'output_encoder_error': (quick_status & (1 << 1)) != 0,
    'calibration_encoder_error': (quick_status & (1 << 2)) != 0,
    'mosfet_bridge_error': (quick_status & (1 << 3)) != 0,
    'hardware_errors': (quick_status & (1 << 4)) != 0,
    'communication_errors': (quick_status & (1 << 5)) != 0,
    'motion_errors': (quick_status & (1 << 6)) != 0
}
self.get_logger().info(error_bits)

errors_detected = False

for key, val in error_bits.items():
    if val:
        self.get_logger().error(f"ID {id}: {key}")
        errors_detected = True

[PawelKorsak]

Thank you for the submission.

Could you share the code fragment that is performed after the begin() function? It will be useful in solving this.

This message might be displayed because of the fact that candle is operating in update mode which ignores all the config commands and allows only operations that are performed on md80 reference objects. When issuing config commands while in update mode the message [CANDLE] Did not receive response from CANdle! [FAILED] might come up.

To know for certain what is happening I would need to look at the code that is performed when the error comes up.

[missxa]

thanks @PawelKorsak for the reply. all config happens before .begin() is called. after that there's a ROS subscriber that sets the position targets and a ROS timer that controls the motor via self.candle.md80s[i].setTargetPosition. see the code below.

note, I can only reproduce the issue when the host machine (Jetson Orin) hardware is power cycled. when doing a soft reboot (e.g. sudo reboot) on Jetson, the issue does not appear. this issue is quite critical, since your motors are integrated on a robot and they fail every time the robots boots on and your API does not allow to automatically detect this failure. help and pointers will be appreciated!

self.candle = pyCandle.Candle(pyCandle.CAN_BAUD_1M, True)

ids = self.candle.ping()
if len(ids) == 0:  # If no drives found -> quit
    sys.exit("EXIT FAILURE")
for m in self.motor_ids:
    if m not in ids:
        self.get_logger().error(f"Motor {m} is not on the CANdle bus")

for id in self.motor_ids:
    self.candle.addMd80(id)

for id in self.motor_ids:
    self.candle.writeMd80Register(id, pyCandle.Md80Reg_E.maxAcceleration, 30.0)
    self.candle.writeMd80Register(id, pyCandle.Md80Reg_E.maxDeceleration, 30.0)
    self.candle.writeMd80Register(id, pyCandle.Md80Reg_E.maxVelocity, 10.0)
    self.candle.writeMd80Register(id, pyCandle.Md80Reg_E.maxTorque, 20.0)

    self.candle.writeMd80Register(id, pyCandle.Md80Reg_E.positionWindow, 0.01)
    self.candle.writeMd80Register(id, pyCandle.Md80Reg_E.velocityWindow, 0.01)
    self.candle.writeMd80Register(id, pyCandle.Md80Reg_E.profileAcceleration, 20.0)
    self.candle.writeMd80Register(id, pyCandle.Md80Reg_E.profileDeceleration, 20.0)
    self.candle.writeMd80Register(id, pyCandle.Md80Reg_E.profileVelocity, 20.0)
    self.candle.writeMd80Register(id, pyCandle.Md80Reg_E.quickStopDeceleration, 200.0)

    self.candle.writeMd80Register(id, pyCandle.Md80Reg_E.motorPosPidKp, 10.0)
    self.candle.controlMd80SetEncoderZero(id)
    self.candle.controlMd80Mode(id, pyCandle.POSITION_PID)
    for md in self.candle.md80s:
        md.setTargetPosition(0)
    
    self.candle.controlMd80Enable(id, True)

self.candle.begin()

self.get_logger().info("Started candle")

self.timer = self.create_timer(1.0 / UPDATE_RATE, self.update_targets)

self.subscription = self.create_subscription( # just a ROS2 subscriber that sets target positions for both motors
    JointState,
    '/mab/target/joint_state',
    self.joint_state_callback,
    1
)

def update_targets(self):
    for motor_idx, desired_target in self.desired_targets.items():
            <...>
            self.candle.md80s[motor_idx].setTargetPosition(self.current_targets[motor_idx])

and the resulting log:

[CANDLE] CANdle library version: v3.4.1
[CANDLE] Device firmware version: v2.2.1
[CANDLE] CANdle at /dev/ttyACM0, ID: 0xac12a51e374c1f30 ready (USB)
[CANDLE] Starting pinging drives at baudrate: 1M
[CANDLE] Found drives.
[CANDLE] 1: ID = 144 (0x90)
[CANDLE] 2: ID = 145 (0x91)
[CANDLE] Added MD80 with ID: 145  [OK]
[CANDLE] Added MD80 with ID: 144  [OK]
[CANDLE] Setting new zero position successful at ID: 145  [OK]
[CANDLE] Setting control mode successful at ID: 145  [OK]
[CANDLE] Enabling successful at ID: 145  [OK]
[CANDLE] Setting new zero position successful at ID: 144  [OK]
[CANDLE] Setting control mode successful at ID: 144  [OK]
[CANDLE] Enabling successful at ID: 144  [OK]
[CANDLE] Beginnig auto update loop mode  [OK]
[INFO] [0000000070.266630784] [cubemars_ctl]: Started candle
[INFO] [0000000070.289371584] [cubemars_ctl]: Shoulders are initialized
[CANDLE] Did not receive response from CANdle!  [FAILED]