detect_and_segment

[HrishikeshVish]

I ran into two issues.

I am trying to reproduce the tea-pouring example used in the paper. I passed an image from the PhysGENBench into the stage1 config file.

EXP_NAME="initial_setup"
PROMPT="The video shows a close-up of a clear glass being filled with tea. The tea is being poured from above, and we can see the stream of tea
hitting the bottom of the glass and causing ripples and splashes. The background is a plain white surface, which contrasts with the transparency of the glass and
the clarity of the tea. The glass is cylindrical in shape and appears to be of a standard size for a drinking glass.
initial_boxes: [{'id': 0, 'name': 'tea', 'box': [241.8, 335.3, 165, 88]}]"
DATA_ROOT="data"
FIRST_FRAME_PATH="PhyGenBench/frame0.png"
GROUNDING_MODEL_PATH="IDEA-Research/grounding-dino-tiny"
SAM2_CHECKPOINT="GroundedSAM2/checkpoints/sam2.1_hiera_large.pt"

python -m VLIPP.stage1.stage1 --prompt "$PROMPT"
--data_root "$DATA_ROOT"
--first_frame_path "$FIRST_FRAME_PATH"
--exp_name "$EXP_NAME"
--grounding_model_path "$GROUNDING_MODEL_PATH"
--sam2_checkpoint "$SAM2_CHECKPOINT"

Where do I obtain grounding model?

The first error I got was: using image API for GPT 4o

messages[-1]['content'][0]['image_url'] = {"url": f"data:image/png;base64,{first_frame_b64_str}"}

I had to change all "image" keys to "image_url". Is this expected?

Issue #2:
data_dict = detect_and_segment(data_dict, output_root=output_root, sam2_model=sam2_model, sam2_predictor=sam2_predictor, processor=processor, grounding_model=grounding_model)

detect_and_segment does not have the right function signature : def detect_and_segment(info_json, output_root, sam2_predictor=None, processor=None, grounding_model=None, DEVICE="cuda"):

[Madaoer]

Hi, thanks for your comment! Please see my responses below:

1. You can download the grounding model by following the instructions provided in the link.

2. This might be caused by different versions of OpenAI. I believe It won’t affect the results much.

3. I’ve fixed this issue along with some other bugs. Thank you for pointing it out^^

[HrishikeshVish]

There might be another bug in compute iou
xa1, ya1, xa2, ya2 = x1, y1, x1 + w1, y1 + h1
xb1, yb1, xb2, yb2 = x2, y2, x2 + w2, y2 + h2
These values aren't defined

Additionally, image_size = (720, 480) --> This is causing issues.

I'm trying to run stage1 for the pouring example. Exactly how should I run it to reproduce the results in the paper? What is the input, prompt and output?
FIRST_FRAME_PATH="PhyGenBench/frame0.png" -> Is this correct?

When I run it, I see a bounding box expanding on the right corner of the image, possibly because my sizes are incorrect.
<PIL.Image.Image image mode=RGB size=1960x836 at 0x77806D6A85B0>
MERGED IMAGE: (836, 1960, 3) MASK (836, 1960) (0, 255, 0)

[HrishikeshVish]

In motion_animation.py, should the bounding box be defined like this:

Line 226: #x1, y1, x2, y2 = bbox
New definition: x1, y1, w1, h1 = bbox

[Madaoer]

Hi, sorry for late reply. Please see my responses below:

There might be another bug in compute iou xa1, ya1, xa2, ya2 = x1, y1, x1 + w1, y1 + h1 xb1, yb1, xb2, yb2 = x2, y2, x2 + w2, y2 + h2 These values aren't defined

Thanks. I will update.

Additionally, image_size = (720, 480) --> This is causing issues.

The input first frame should be (720, 480) (width, height). If not, please resize it first.

I'm trying to run stage1 for the pouring example. Exactly how should I run it to reproduce the results in the paper? What is the input, prompt and output? FIRST_FRAME_PATH="PhyGenBench/frame0.png" -> Is this correct?

The input to our framework should be the first frame and a prompt. FIRST_FRAME_PATH=PhyGenBench/frame0.png is your file path. For example, if you have a folder named PhyGenBench, there should be a file named frame0.png inside it.

When I run it, I see a bounding box expanding on the right corner of the image, possibly because my sizes are incorrect. <PIL.Image.Image image mode=RGB size=1960x836 at 0x77806D6A85B0> MERGED IMAGE: (836, 1960, 3) MASK (836, 1960) (0, 255, 0)

As mentioned, please resize the first frame to (720, 480) (width, height).

In motion_animation.py, should the bounding box be defined like this:
Line 226: #x1, y1, x2, y2 = bbox
New definition: x1, y1, w1, h1 = bbox

Thanks. I will update.

[pushchar]

Hi, thanks for sharing your work!

Just wanted to follow up on this, will you be committing the bug fixes soon?