The issue of trajectory success evaluation in WebArena

[ZLZLGe]

While examining the trajectories, I noticed that the action fields contain a thinking component. Could you clarify whether this affects task evaluation? For example, if the correct answer appears in the thinking section but not in the actual action output, is it still considered a successful completion, or is the evaluation based solely on the action content?

[jmzhoulab]

Task evaluation is solely based on the content within the <action> tag, not on the thinking/reasoning section. If the correct answer appears only in the thinking section but not in the actual action output, it will not be considered a successful completion.

For tasks that require text responses (e.g., answering questions, providing information), the evaluation is based on the parameter content of the send_msg_to_user action, not the thinking section. Even if the correct answer appears in the <think> tag, it must be included in the send_msg_to_user action to be considered for evaluation.

Example:

<think>
The task is to identify the top-1 best-selling product in 2022. I'm on the Bestsellers Report page under Reports > Products. The filters are correctly set: Period = Year, From = 1/1/22, To = 12/31/22, and the report has been shown. The results table lists products with their order quantities for the interval 2022. The first product row shows "Quest Lumaflex™ Band" with an order quantity of 5, followed by other products with lower quantities (4 and 3). Therefore, the top-1 best-selling product for 2022 is "Quest Lumaflex™ Band". I will return the exact product name, preserving its trademark symbol as required.
</think>
<action>
```send_msg_to_user('Quest Lumaflex™ Band')```
</action>

[ZLZLGe]

Task evaluation is solely based on the content within the <action> tag, not on the thinking/reasoning section. If the correct answer appears only in the thinking section but not in the actual action output, it will not be considered a successful completion.

For tasks that require text responses (e.g., answering questions, providing information), the evaluation is based on the parameter content of the send_msg_to_user action, not the thinking section. Even if the correct answer appears in the <think> tag, it must be included in the send_msg_to_user action to be considered for evaluation.

Example:

<think>
The task is to identify the top-1 best-selling product in 2022. I'm on the Bestsellers Report page under Reports > Products. The filters are correctly set: Period = Year, From = 1/1/22, To = 12/31/22, and the report has been shown. The results table lists products with their order quantities for the interval 2022. The first product row shows "Quest Lumaflex™ Band" with an order quantity of 5, followed by other products with lower quantities (4 and 3). Therefore, the top-1 best-selling product for 2022 is "Quest Lumaflex™ Band". I will return the exact product name, preserving its trademark symbol as required.
</think>
<action>
```send_msg_to_user('Quest Lumaflex™ Band')```
</action>

Thanks for your reply. Best regards.