Questions on Video Representation and Inference in VITRA

[UEFI-code]

1. 2D->3D, Ground-Truth Data
   In the paper, it seems to be stated that the “raw videos” do not contain labeling information.
   I would like to ask: are the raw videos monocular 2D videos captured by a single camera? If so, do you then use some depth estimation algorithms to infer 3D information from them?

2. "Brain" doesn't participate in autoregressive inference

        output_hs, inputs_masks = self.prepare_vlm_features(
            pixel_value,
            input_ids,
            attention_mask,
            current_state_mask,
            current_state,
            fov,
            use_cache=use_cache,
        )
        # handle multiple samples for one input
        samples, _ = self._forward_act_model(
            vlm_features = output_hs,
            attention_mask = inputs_masks,
            action_masks = x_mask,
            current_state = current_state,
            current_state_mask = current_state_mask,
            mode = "eval",
            repeated_diffusion_steps = sample_times,
            cfg_scale = cfg_scale,
            use_ddim = use_ddim,
            num_ddim_steps = num_ddim_steps,
        )
        action_np = samples.cpu().numpy() * x_mask.cpu().numpy()    # sample_times x T x D
        return action_np

3. Open-Loop Robotic Ctrl, with NO feedback
   Robot break something -> Model never know

[arnoldland]

1. EgoExo4D is captured using a multi-view camera setup. For the other datasets, the raw videos are collected using a monocular camera. In these cases, we employ pretrained models to perform depth estimation and infer 3D information. Please refer to Section 3.1 (3D Motion Labeling) of the paper for detailed descriptions.

2. During inference, the VLM does not perform autoregressive generation in the same manner as OpenVLA. Instead, it generates outputs jointly with the action expert. Further details can be found in Section 4.1 (VLA Model Design) of the paper.

3. The control strategy is a commonly adopted strategy in VLA-based methods. Within an action chunk, the model does not receive feedback for closed-loop control. However, between action chunks, the model re-observes the current image and generates new actions based on the updated visual feedback.

[UEFI-code]

Thank you for your reply.

I still have some questions

• Does one chunk ref to one action (ref to 8 frames in the paper)?
• Seems VLM learned to estimate depth? You input only one 2D frame during prediction?

[arnoldland]

1. In our VLA, an action chunk consists of 16 actions, meaning that the VLA predicts 16 future actions at once. The action chunk is independent of the number of frames fed into GPT during instruction labeling; these are two separate and unrelated design choices.

2. In our VLA, we indeed use only a single 2D image as input for action prediction, which is also a common practice in VLA-based approaches. Through training on our dataset, the VLA (or the VLM part) acquires a certain capability for depth estimation.

[UEFI-code]

1. In our VLA, an action chunk consists of 16 actions, meaning that the VLA predicts 16 future actions at once. The action chunk is independent of the number of frames fed into GPT during instruction labeling; these are two separate and unrelated design choices.

16 actions means 16 ctrl steps of the robotic arm?

2. In our VLA, we indeed use only a single 2D image as input for action prediction, which is also a common practice in VLA-based approaches. Through training on our dataset, the VLA (or the VLM part) acquires a certain capability for depth estimation.

Is 2D -> 3D depth estimation always reliable in real-world?
Would't it overfit something like "near is big and far is small"?

[arnoldland]

yes, '16 actions' means 16 ctrl steps of the robotic arm and hand.

In the real world, monocular depth estimation is not always reliable and often suffers from ambiguity. However, pretrained depth estimation models, such as MoGe-2, have acquired strong generalization capabilities from large-scale data, rather than simply overfitting to “near objects appear larger and far objects appear smaller”. In our 3D annotation pipeline, the priors provided by MoGe-2 are further refined by integrating visual SLAM, resulting in more accurate annotations. Training the VLA on data constructed using such models, to some extent, distills their knowledge into the VLA.

[UEFI-code]

yes, '16 actions' means 16 ctrl steps of the robotic arm and hand.

How much range of motion is controlled at each step? For example 15º ?

In the real world, monocular depth estimation is not always reliable and often suffers from ambiguity. However, pretrained depth estimation models, such as MoGe-2, have acquired strong generalization capabilities from large-scale data, rather than simply overfitting to “near objects appear larger and far objects appear smaller”. In our 3D annotation pipeline, the priors provided by MoGe-2 are further refined by integrating visual SLAM, resulting in more accurate annotations. Training the VLA on data constructed using such models, to some extent, distills their knowledge into the VLA.

It is interested to discuss depth estimation models.
I may launch a human experiment of single-eye depth estimation.
near-small-obj vs far-big-obj

Ps: if depth-estimation so accurate, ToF camera company will lose money :)

[arnoldland]

It is interested to discuss depth estimation models. I may launch a human experiment of single-eye depth estimation. near-small-obj vs far-big-obj

Ps: if depth-estimation so accurate, ToF camera company will lose money :)

Yes, we agree. Even humans cannot reliably infer depth from a single view. Depth estimation models are therefore expected to make plausible guesses under ambiguity, rather than perfectly resolving it. Along the same lines, the human-data pretraining in our VLA aims to teach the model to produce a reasonable pose, capturing how the agent should roughly move given the instruction, rather than metrically accurate motion.