Skip to content

Coordination inaccuracy issue #83

@agn-7

Description

@agn-7

Gemini 2.5 Computer Use Preview exhibits inconsistent behavior regarding the accuracy of click_at coordinates after a certain number of iterations. While the output_text shows a reasoning with valid coordinates within the normalized 0-1000 range, the function_call click_at action often provides coordinates exceeding this maximum, leading to incorrect clicks. This issue seems to manifest after approximately 15 iterations. I am attaching a llm logs of the output_text and function_call from a recent instance where this occurred.

Please provide any additional information required for your investigation.

{
  "timestamp": "2025-10-17T08:17:47.122Z",
  "success": true,
  "id": "gcua_1760689067122_nhvkogrhj",
  "model": "gemini-2.5-computer-use-preview-10-2025",
  "output": [
    {
      "type": "output_text",
      "text": "Pressing 'escape' also did not close the modal. This is strange.\nThe modal \"Apply to Explore Group\" is preventing me from interacting with the page.\nMaybe I need to click the \"Dismiss\" button at the bottom right (y=940, x=720) if there is one, or try clicking outside the modal area again, very far from it. Or maybe the 'X' (y=190, x=720) again.\nIt looks like the 'X' is the only close option. I'll try it one more time, very carefully aiming for its center."
    },
    {
      "type": "computer_call",
      "call_id": "call_1760689067122_movb2b9yu",
      "action": {
        "type": "click_at",
        "y": 139,
        "x": 1922
      }
    }
  ],
  "usage": {
    "input_tokens": 9779,
    "output_tokens": 157
  },
  "finishReason": "STOP"
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions