When you were testing ScanQA and SQA3D, how did you extract key objects?

[alivecat05]

Please allow me to show my appreciation to your brilliant work?

But I have a question:When you directly test ScanQA and SQA3D, how do you extract the key objects? The original problems of these two datasets seem unlikely to extract using the rule-based approach.

[SARIHUST]

Thank you for your interest in our work, and really sorry for the late reply.

Regarding your question, it is actually possible to extract the key objects directly from the questions in both ScanQA and SQA3D. For example:

• ScanQA:

{
    "scene_id": "scene0011_00",
    "obj_id": 17,
    "prompt": "What color is the tv above a thin table? Answer the question using a single word or phrase.",
    "ref_captions": ["black", "black"],
    "qid": 11
}

• SQA3D:

{
    "scene_id": "scene0699_00",
    "obj_id": 0,
    "prompt": "I grabbed the blanket on the floor and to my left was the door. Are the closet doors in front of me open or closed? Answer the question using a single word or phrase.",
    "ref_captions": ["open"],
    "qid": 60
}

As you can see, the questions already contain object mentions (e.g., “tv”, “table”), so we can extract these as key objects and then match them with the object list that the detection/segmentation model predicts.

[alivecat05]

Thank you for your interest in our work, and really sorry for the late reply.

Regarding your question, it is actually possible to extract the key objects directly from the questions in both ScanQA and SQA3D. For example:

• ScanQA:

{
"scene_id": "scene0011_00",
"obj_id": 17,
"prompt": "What color is the tv above a thin table? Answer the question using a single word or phrase.",
"ref_captions": ["black", "black"],
"qid": 11
}

• SQA3D:

{
"scene_id": "scene0699_00",
"obj_id": 0,
"prompt": "I grabbed the blanket on the floor and to my left was the door. Are the closet doors in front of me open or closed? Answer the question using a single word or phrase.",
"ref_captions": ["open"],
"qid": 60
}
As you can see, the questions already contain object mentions (e.g., “tv”, “table”), so we can extract these as key objects and then match them with the object list that the detection/segmentation model predicts.

Directly use Regular Expression or prompt a llm?