Some questions about this code

[maobenz]

Firstly, thank you very much for releasing the code!

I have several questions regarding the camera and pose settings:

1. How are the camera poses for EPIC-Kitchens obtained? As far as I know, the EPIC-Kitchens dataset does not provide camera poses directly. Did you use EPIC Fields, which includes camera pose information? I have tried the camera poses of Epic fields, the camera poses seem wrong to render the robot.

2. How are the camera world coordinates aligned with the Mujoco world coordinates? From the code, it seems that the extrinsics are used to transform poses from the camera frame to the robot Mujoco frame. However, are the camera world coordinates assumed to be the same as the Mujoco world coordinates? My understanding is that the extrinsics of the camera originate from the real-world setup, not from the Mujoco environment.

3. How is the robot base position determined? I noticed the code uses a default value:

DEFAULT_ROBOT_BASE_POS = np.array([-0.56, 0, 0.912])

Could you explain the rationale for choosing this specific position?

4. Regarding p_marker_ee, how is this quantity obtained?

Thank you very much for your time and clarification!

[JiayingFang]

Hi, thanks for your interest in our project!

1. Yes, we did use the camera extrinsics in EPIC fields in our project, but these camera extrinsics were only used to filter out frames with large camera motions, and this part of the code is not included in this release. The camera poses you see in phantom/camera folder are different, they describe the relative pose between the camera and the robot base in our setup.

2. You are correct that the camera extrinsics in phantom/camera are used to transform poses from the camera frame to the robot frame. The world coordinate in Mujoco environment does not really matter, as long as you set up the Mujoco environment in a way that the relative pose between the camera and robot base in the Mujoco environment is the same as their relative pose in real-world setting.

3. This value only depends on how you set up your Mujoco environment. The main purpose of using DEFAULT_ROBOT_BASE_POS here is to do the conversion between action in robot frame and action in world frame in our single_arm Mujoco environment. Of course this only works when the transformation between the robot coordinate and world coordinate is simply this DEFAULT_ROBOT_BASE_POS position change (which is the case in our single arm Mujoco environment).

4. I think you'll be able to find the answer here, we modified this code for camera calibration: https://github.com/facebookresearch/fairo/blob/main/perception/sandbox/eyehandcal/src/eyehandcal

Hope this helps! Feel free to let me know if you have any other questions!

[maobenz]

Thanks a lot!

I find it very difficult to determine the correct value for DEFAULT_ROBOT_BASE_POS. I selected an Epic Fields video clip and used your code, but the rendered quality was very poor. Could you kindly help us check?

Here is the video link: Google Drive
.

I used your provided camera poses and use my own intrinsics. However, when I keep the default robot base position, the output is completely black and the error is very large. If I manually adjust the robot base position, it is still hard to find a proper setting that renders correctly. Do you have any advice on how to adjust the robot base position? I also tried changing the position in your demo, but that failed as well. Does this mean the base position must be adjusted for each specific video?

Another question is about depth. How can we acquire the depth of the hands? I noticed that without hand depth, it seems incorrect to estimate the 3D hand pose.

Finally, do you have the plan to release the edited videos in your paper?

Thank you very much!

[JiayingFang]

Hi!

This code should work out-of-the-box for Epic-Kitchen videos, the videos we provided as sample videos in data/epic are already videos extracted from Epic-Kitchens (these videos are downloaded during installation of the code). Have you had a chance to try those?

DEFAULT_ROBOT_BASE_POS is only used here for single arm environment, which is not the one we are using for Epic-Kitchen videos. For Epic-Kitchen videos, we keep using twin_bimanual_robot.py. For Epic-Kitchen videos, could you try adding the video you selected from Epic-Kitchen (and also the hand detection label extracted from Epic-Kitchens dataset) into data/epic folder and then running this command we provided in README:
python process_data.py demo_name=epic data_root_dir=../data/raw processed_data_root_dir=../data/processed mode=all --config-name=epic

For changing camera intrinsics, currently our code for Phantom setting (single arm, in-lab collected) only works for resolution HD1080. We currently don't have enough time to update the code, but feel free to modify it to support different resolutions, should be a straightforward change.

In Phantom, we use the depth from ZED2 camera; in Masquerade, we do not use depth because there is no depth info, therefore, the edited videos we get are not "perfect". One reason we don’t perform zero-shot transfer in Masquerade is that the videos have depth issues and are not "perfect".

We've provided some sample videos in data folder. Both Marion and I just graduated and don't have enough time to release the whole dataset of edited videos recently, but we will keep in mind this request. We'll let you if we decide to release the whole dataset of edited videos.

[maobenz]

Thank you very much!

We were successful in using other clips from Epic Kitchen for rendering.

I have a few questions:

1. In phantom/camera, you mention that the pose differs from the camera poses in Epic Fields, and that it describes the relative pose between the camera and the robot base in your setup. Could you clarify how you obtain the transformations between the camera and the robot base? Although you note that “the world coordinate in the Mujoco environment does not really matter, as long as you set up the Mujoco environment so that the relative pose between the camera and robot base matches their relative pose in the real-world setting”, does this mean we can try different poses and the results will always be correct?

2. In twin_bimanual_robot, how is BASE_T_1 determined? Does this mean that the predicted hand poses need to be processed through two transformations—first from camera to robot, and then from robot to Epic Kitchen?

3. If I want to use other datasets such as Ego4D, do you have suggestions on how to adjust the transformations and hyperparameters? Many parameters appear to be fixed for the Epic Kitchen dataset, so I wonder what would need to be adapted for other datasets.

Thank you again for your guidance.

[edisiondyli]

@maobenz How are the results when running on EPIC-KITCHENS? When I run it, I often find that the hand bounding boxes are accurate, but the hand pose cannot be detected and an edge hand error is reported. Have you encountered a similar situation?

[JiayingFang]

Thank you very much!

We were successful in using other clips from Epic Kitchen for rendering.

I have a few questions:

1. In phantom/camera, you mention that the pose differs from the camera poses in Epic Fields, and that it describes the relative pose between the camera and the robot base in your setup. Could you clarify how you obtain the transformations between the camera and the robot base? Although you note that “the world coordinate in the Mujoco environment does not really matter, as long as you set up the Mujoco environment so that the relative pose between the camera and robot base matches their relative pose in the real-world setting”, does this mean we can try different poses and the results will always be correct?
2. In twin_bimanual_robot, how is BASE_T_1 determined? Does this mean that the predicted hand poses need to be processed through two transformations—first from camera to robot, and then from robot to Epic Kitchen?
3. If I want to use other datasets such as Ego4D, do you have suggestions on how to adjust the transformations and hyperparameters? Many parameters appear to be fixed for the Epic Kitchen dataset, so I wonder what would need to be adapted for other datasets.

Thank you again for your guidance.

It's great to hear you were successful in using other epic kitchens clips for rendering!

Regarding the questions:

1. We attached an Aruco marker on the end-effector of the robot. It's a typical eye-on-base camera calibration.

2. We first manually selected a base pose for one of the arms (for example, arm A) of the bimanual robot in mujoco, and then we can obtain the base pose of the other robot arm (arm B) and the camera pose in simulation using:
   i. the relative pose between arm A's base and camera in real world,
   ii. the relative pose between arm B's base and the camera in real world.
   These two relative poses can be obtained using camera calibration.

3. I would recommend first double-check image resolutions and camera intrinsics used in Ego4D. As for extrinsics settings, that shouldn't be related to the dataset (i.e. Epic kitchens or Ego4D).

Hope this helps :-)

[maobenz]

@maobenz How are the results when running on EPIC-KITCHENS? When I run it, I often find that the hand bounding boxes are accurate, but the hand pose cannot be detected and an edge hand error is reported. Have you encountered a similar situation?

yes, there are some bad cases

[edisiondyli]

@JiayingFang Hi Jiaying, we encountered a couple of issues we would like to clarify when working on reproducing the robot inpainting pipeline:

1. When running the pipeline with the sample data provided in your repository, we noticed that the inpainting results in the first half of the sequence (both arm and hand inpainting) do not look as expected. Could you please advise whether this is a known issue or if there are additional configuration steps we may have missed?

2. We are also trying to apply the pipeline to other EPIC-Kitchens video clips. To better align with the annotations, we split the long videos into shorter clips based on narration segments. However, we are unsure how to correctly handle the corresponding hand_det.pkl files:

• Should the 'hand_det.pkl' be split strictly according to the absolute frame indices from the original video?

• Or should we re-index each narration-based segment starting from zero?

• We noticed that incorrect splitting or indexing leads to missing detections or “zero objs” errors in the pipeline.

We would greatly appreciate your guidance on how to properly align the narration-based clips with the hand_det.pkl files. Thank you very much for your time and for sharing your excellent work.

[JiayingFang]

Hi @edisiondyli , really sorry for the late reply.

1. Sadly I'm unable to see your inpainting results now, I don't have the access to your link. Let me know if this is still an issue for you.
2. 'hand_det.pkl' needs to be split according to the absolute frame indices from the original video.

Hope this helps!

[3587jjh]

Hi @JiayingFang , thank you for your previous replies and it helped a lot for furthur understanding the code.
I have a small follow-up question about the camera intrinsics used for Epic-Kitchens.

In phantom/camera/camera_intrinsics_epic.json, there is a single set of intrinsics that appears to be used for all Epic-Kitchens clips. I was wondering how these values were obtained.

Since EPIC-Fields provides per-video intrinsic estimates and EPIC-Kitchens-100 includes recordings from both GoPro Hero 5 and Hero 7, how should we interpret using one shared intrinsics file for all Epic-Kitchens clips?

I also noticed that the values in camera_intrinsics_epic.json are numerically very close to the calibrated intrinsics of the robot camera (defined in phantom/camera/camera_extrinsics_ego_bimanual_shoulders.json). Are the Epic intrinsics intended to match the robot camera, or do they correspond to the actual Epic-Kitchens camera?

I would greatly appreciate any guidance you could provide.
Thank you very much for your time.

[JiayingFang]

Hi @3587jjh , sorry for the confusion, we just use one intrinsics setting from EPIC-Fields because only one camera type is used in the EPIC dataset and the intrinsics from the same camera type are generally similar. However it'll definitely be better to use the estimated intrinsic per-video from EPIC-Fields.
I think the values in camera_intrinsics_epic.json are not close to that in camera_extrinsics_ego_bimanual_shoulders.json? The two intrinsics do not need to match because we are doing co-training. Also you could ignore h_fov and d_fov in the intrinsics, these are never used.