使用api调用qwen3vl测试结果非常差

[coco515alt]

api调用的qwen3vl准确率大概只有1%，我将GPT5中的url、apikey和model name换成了qwen3vl，prompt没变。由于qwen3vl返回的结果是原始坐标我多调用了一次qwen3vl询问它看到的图像大小然后手动做了归一化但是结果偏差非常大，请问作者我的测试流程是否有问题？我还注意到排行榜上面gpt系列的准确率也很低，但是qwen2.5vl的准确率非常高。

[likaixin2000]

Hi, thanks for using ScreenSpot-Pro. Qwen3-VL uses relative coordinates similar to Qwen2-VL. Please see this issue: QwenLM/Qwen3-VL#1485

GPT can't do grounding, so the low score is expected.

[coco515alt]

Thank you for your response. I’ve successfully reproduced the results of Qwen3-VL, with an accuracy of around 51%. Have you considered adding a test script for Qwen3-VL to the project? If needed, I could contribute my own test script.

[likaixin2000]

I am glad to hear that!

I have plans to add the script, but haven't started yet. I'd be very grateful if you could submit a PR.

[Cescfangs]

Thank you for your response. I’ve successfully reproduced the results of Qwen3-VL, with an accuracy of around 51%. Have you considered adding a test script for Qwen3-VL to the project? If needed, I could contribute my own test script.

@coco515alt hey，may I ask how do you eval qwen3-vl? I was using the computer-use prompt:

[
 {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": "You are a helpful assistant."
            },
            {
                "type": "text",
                "text": "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"type\": \"function\", \"function\": {\"name\": \"computer_use\", \"description\": \"Use a mouse and keyboard to interact with a computer, and take screenshots.\\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.\\n* The screen's resolution is 1000x1000.\\n* Whenever you intend to move the cursor to click on an element like an icon, you should consult a screenshot to determine the coordinates of the element before moving the cursor.\\n* If you tried clicking on a program or link but it failed to load, even after waiting, try adjusting your cursor position so that the tip of the cursor visually falls on the element that you want to click.\\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges.\", \"parameters\": {\"properties\": {\"action\": {\"description\": \"The action to perform. The available actions are:\\n* `key`: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\\n* `type`: Type a string of text on the keyboard.\\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `left_click`: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `left_click_drag`: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\\n* `right_click`: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `middle_click`: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `double_click`: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\\n* `triple_click`: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it's the closest action).\\n* `scroll`: Performs a scroll of the mouse scroll wheel.\\n* `hscroll`: Performs a horizontal scroll (mapped to regular scroll).\\n* `wait`: Wait specified seconds for the change to happen.\\n* `terminate`: Terminate the current task and report its completion status.\\n* `answer`: Answer a question.\", \"enum\": [\"key\", \"type\", \"mouse_move\", \"left_click\", \"left_click_drag\", \"right_click\", \"middle_click\", \"double_click\", \"triple_click\", \"scroll\", \"hscroll\", \"wait\", \"terminate\", \"answer\"], \"type\": \"string\"}, \"keys\": {\"description\": \"Required only by `action=key`.\", \"type\": \"array\"}, \"text\": {\"description\": \"Required only by `action=type` and `action=answer`.\", \"type\": \"string\"}, \"coordinate\": {\"description\": \"(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to.\", \"type\": \"array\"}, \"pixels\": {\"description\": \"The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by `action=scroll` and `action=hscroll`.\", \"type\": \"number\"}, \"time\": {\"description\": \"The seconds to wait. Required only by `action=wait`.\", \"type\": \"number\"}, \"status\": {\"description\": \"The status of the task. Required only by `action=terminate`.\", \"type\": \"string\", \"enum\": [\"success\", \"failure\"]}}, \"required\": [\"action\"], \"type\": \"object\"}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>"
            }
        ]
 },
     {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": ""
                }
            },
            {
                "type": "text",
                "text": "open the third issue"
            }
        ]
    }
]

parse coordinates from response and normalize to 0-1(x/1000, y/1000), the result was really bad

[coco515alt]

@Cescfangs

Thank you for your response. I’ve successfully reproduced the results of Qwen3-VL, with an accuracy of around 51%. Have you considered adding a test script for Qwen3-VL to the project? If needed, I could contribute my own test script.感谢您的回复。我已经成功复现了 Qwen3-VL 的结果，准确率约为 51%。您是否考虑过在项目中添加 Qwen3-VL 的测试脚本？如果需要，我可以贡献我自己的测试脚本。

@coco515alt hey，may I ask how do you eval qwen3-vl? I was using the computer-use prompt:嘿，请问一下你是怎么评估 qwen3-vl 的？我使用的是电脑使用提示：

[
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a helpful assistant."
},
{
"type": "text",
"text": "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n\n{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to interact with a computer, and take screenshots.\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.\n* The screen's resolution is 1000x1000.\n* Whenever you intend to move the cursor to click on an element like an icon, you should consult a screenshot to determine the coordinates of the element before moving the cursor.\n* If you tried clicking on a program or link but it failed to load, even after waiting, try adjusting your cursor position so that the tip of the cursor visually falls on the element that you want to click.\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:\n* key: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\n* type: Type a string of text on the keyboard.\n* mouse_move: Move the cursor to a specified (x, y) pixel coordinate on the screen.\n* left_click: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\n* left_click_drag: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\n* right_click: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\n* middle_click: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\n* double_click: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\n* triple_click: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it's the closest action).\n* scroll: Performs a scroll of the mouse scroll wheel.\n* hscroll: Performs a horizontal scroll (mapped to regular scroll).\n* wait: Wait specified seconds for the change to happen.\n* terminate: Terminate the current task and report its completion status.\n* answer: Answer a question.", "enum": ["key", "type", "mouse_move", "left_click", "left_click_drag", "right_click", "middle_click", "double_click", "triple_click", "scroll", "hscroll", "wait", "terminate", "answer"], "type": "string"}, "keys": {"description": "Required only by action=key.", "type": "array"}, "text": {"description": "Required only by action=type and action=answer.", "type": "string"}, "coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to.", "type": "array"}, "pixels": {"description": "The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by action=scroll and action=hscroll.", "type": "number"}, "time": {"description": "The seconds to wait. Required only by action=wait.", "type": "number"}, "status": {"description": "The status of the task. Required only by action=terminate.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}}}\n\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{"name": , "arguments": }\n</tool_call>"
}
]
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": ""
}
},
{
"type": "text",
"text": "open the third issue"
}
]
}
]
parse coordinates from response and normalize to 0-1(x/1000, y/1000), the result was really bad从响应中解析坐标并将其标准化为 0-1（x/1000，y/1000），结果非常糟糕

This is a grounding test rather than a computer-use test. The model is not provided with any tools, so the prompt should emphasize that function calls must not be used. You can refer to the prompt the repository author designed for GPT-5. Below is the prompt I designed—hope it’s helpful.response = self.client.chat.completions.create(
model=self.model_name,
messages=[
{ "role": "system",
"content": [{"type": "text", "text": "You are an expert in using electronic devices and interacting with graphic interfaces. You should not call any external tools."}],
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{base64_image}",
},
},
{
"type": "text",
"text": f"Query: {instruction}\nOutput only the coordinate of one point in your response.\nFormat: x,y\n",
},
],
},
],
temperature=self.override_generation_config["temperature"],
max_tokens=2048
)

[Cescfangs]

@Cescfangs

Thank you for your response. I’ve successfully reproduced the results of Qwen3-VL, with an accuracy of around 51%. Have you considered adding a test script for Qwen3-VL to the project? If needed, I could contribute my own test script.感谢您的回复。我已经成功复现了 Qwen3-VL 的结果，准确率约为 51%。您是否考虑过在项目中添加 Qwen3-VL 的测试脚本？如果需要，我可以贡献我自己的测试脚本。

@coco515alt hey，may I ask how do you eval qwen3-vl? I was using the computer-use prompt:嘿，请问一下你是怎么评估 qwen3-vl 的？我使用的是电脑使用提示：
[
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a helpful assistant."
},
{
"type": "text",
"text": "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n\n{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse and keyboard to interact with a computer, and take screenshots.\n* This is an interface to a desktop GUI. You do not have access to a terminal or applications menu. You must click on desktop icons to start applications.\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions. E.g. if you click on Firefox and a window doesn't open, try wait and taking another screenshot.\n* The screen's resolution is 1000x1000.\n* Whenever you intend to move the cursor to click on an element like an icon, you should consult a screenshot to determine the coordinates of the element before moving the cursor.\n* If you tried clicking on a program or link but it failed to load, even after waiting, try adjusting your cursor position so that the tip of the cursor visually falls on the element that you want to click.\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:\n* key: Performs key down presses on the arguments passed in order, then performs key releases in reverse order.\n* type: Type a string of text on the keyboard.\n* mouse_move: Move the cursor to a specified (x, y) pixel coordinate on the screen.\n* left_click: Click the left mouse button at a specified (x, y) pixel coordinate on the screen.\n* left_click_drag: Click and drag the cursor to a specified (x, y) pixel coordinate on the screen.\n* right_click: Click the right mouse button at a specified (x, y) pixel coordinate on the screen.\n* middle_click: Click the middle mouse button at a specified (x, y) pixel coordinate on the screen.\n* double_click: Double-click the left mouse button at a specified (x, y) pixel coordinate on the screen.\n* triple_click: Triple-click the left mouse button at a specified (x, y) pixel coordinate on the screen (simulated as double-click since it's the closest action).\n* scroll: Performs a scroll of the mouse scroll wheel.\n* hscroll: Performs a horizontal scroll (mapped to regular scroll).\n* wait: Wait specified seconds for the change to happen.\n* terminate: Terminate the current task and report its completion status.\n* answer: Answer a question.", "enum": ["key", "type", "mouse_move", "left_click", "left_click_drag", "right_click", "middle_click", "double_click", "triple_click", "scroll", "hscroll", "wait", "terminate", "answer"], "type": "string"}, "keys": {"description": "Required only by action=key.", "type": "array"}, "text": {"description": "Required only by action=type and action=answer.", "type": "string"}, "coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to.", "type": "array"}, "pixels": {"description": "The amount of scrolling to perform. Positive values scroll up, negative values scroll down. Required only by action=scroll and action=hscroll.", "type": "number"}, "time": {"description": "The seconds to wait. Required only by action=wait.", "type": "number"}, "status": {"description": "The status of the task. Required only by action=terminate.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}}}\n\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{"name": , "arguments": }\n</tool_call>"
}
]
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": ""
}
},
{
"type": "text",
"text": "open the third issue"
}
]
}
]
parse coordinates from response and normalize to 0-1(x/1000, y/1000), the result was really bad从响应中解析坐标并将其标准化为 0-1（x/1000，y/1000），结果非常糟糕

This is a grounding test rather than a computer-use test. The model is not provided with any tools, so the prompt should emphasize that function calls must not be used. You can refer to the prompt the repository author designed for GPT-5. Below is the prompt I designed—hope it’s helpful.response = self.client.chat.completions.create( model=self.model_name, messages=[ { "role": "system", "content": [{"type": "text", "text": "You are an expert in using electronic devices and interacting with graphic interfaces. You should not call any external tools."}], }, { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": f"data:image/png;base64,{base64_image}", }, }, { "type": "text", "text": f"Query: {instruction}\nOutput only the coordinate of one point in your response.\nFormat: x,y\n", }, ], }, ], temperature=self.override_generation_config["temperature"], max_tokens=2048 )

@coco515alt thanks, got much better result, from 50.9=>58.0 which is close to official result 60.5

[coco515alt]

@Cescfangs I’m not sure whether you’ve tried UITARS 1.5. I’ve found two different explanations of how it handles coordinates, which has left me quite confused.

[coco515alt]

@Cescfangs By the way, how do you configure Qwen’s parameters and post-processing? Could you share your setup?

[Cescfangs]

@Cescfangs I’m not sure whether you’ve tried UITARS 1.5. I’ve found two different explanations of how it handles coordinates, which has left me quite confused.

UITARS-1.5-7B is based on Qwen25-vl, which using absolute coordinate system, Qwen3-vl switch to relative coordiante system.

@Cescfangs By the way, how do you configure Qwen’s parameters and post-processing? Could you share your setup?

You mean Qwen3-VL?

[coco515alt]

@Cescfangs

@Cescfangs I’m not sure whether you’ve tried UITARS 1.5. I’ve found two different explanations of how it handles coordinates, which has left me quite confused.

UITARS-1.5-7B is based on Qwen25-vl, which using absolute coordinate system, Qwen3-vl switch to relative coordiante system.
Thanks for your explanation.

@Cescfangs By the way, how do you configure Qwen’s parameters and post-processing? Could you share your setup?

You mean Qwen3-VL?
yes

[Cescfangs]

@Cescfangs

@Cescfangs I’m not sure whether you’ve tried UITARS 1.5. I’ve found two different explanations of how it handles coordinates, which has left me quite confused.

UITARS-1.5-7B is based on Qwen25-vl, which using absolute coordinate system, Qwen3-vl switch to relative coordiante system.
Thanks for your explanation.

@Cescfangs By the way, how do you configure Qwen’s parameters and post-processing? Could you share your setup?

You mean Qwen3-VL?
yes

Nothing special with my post-process, check this one QwenLM/Qwen3-VL#1576

[likaixin2000]

Hi guys, here is the official system prompt of Qwen3-VL on ScreenSpot-Pro, please check it out:

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "computer_use", "description": "Use a mouse to interact with a computer.\n* The screen's resolution is {screen_width}x{screen_height}.\n* Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges unless asked.\n* you can only use the left_click and mouse_move action to interact with the computer. if you can't find the element, you should terminate the task and report the failure.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:\n* `mouse_move`: Move the cursor to a specified (x, y) pixel coordinate on the screen.\n* `left_click`: Click the left mouse button with coordinate (x, y).\n* `terminate`: Terminate the current task and report its completion status.", "enum": ["mouse_move", "left_click"], "type": "string"}, "coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=mouse_move` and `action=left_click`.", "type": "array"}, "status": {"description": "The status of the task. Required only by `action=terminate`.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>

Remember to fill in the screen size.

We're working on replicating the results for the community and will share the model file here once it's ready. To clarify, I wasn't the one who tested the official scores for Qwen3-VL on ScreenSpot-Pro, so I'm getting the exact details from my colleagues. Thanks for your patience, and stay tuned!

[Xujxyang]

Hello guys, I am still troubled by this problem. I am also using Qwen3-VL-235B-A22B-Instruct by calling the API, but my test results are as follows:

"overall": {
            "num_correct_action": 308,
            "num_total": 1505,
            "wrong_format_num": 0,
            "action_acc": 0.20465116279069767,
            "text_acc": 0.26680896478121663,
            "icon_acc": 0.10211267605633803
        }

(I simply skipped some test cases that reported request errors during the call.)

This seems to be very different from the results in the paper. Could you please provide a class for Qwen3-VL, or I can provide my code if you are willing to take the time to look at it? best regards.

[likaixin2000]

Hello guys, I am still troubled by this problem. I am also using Qwen3-VL-235B-A22B-Instruct by calling the API, but my test results are as follows:

"overall": {
            "num_correct_action": 308,
            "num_total": 1505,
            "wrong_format_num": 0,
            "action_acc": 0.20465116279069767,
            "text_acc": 0.26680896478121663,
            "icon_acc": 0.10211267605633803
        }

(I simply skipped some test cases that reported request errors during the call.)

This seems to be very different from the results in the paper. Could you please provide a class for Qwen3-VL, or I can provide my code if you are willing to take the time to look at it? best regards.

I haven’t had a chance to start on an implementation yet due to a busy schedule. Could you submit an issue so I can take a look at it?