v0.4.3

🐞 Bug Fixes

• regression: rename AgentToolbox.agent_os back to AgentToolbox.os

Full Changelog: v0.4.2...v0.4.3

v0.4.2

Rerelease of v0.4.1 as something went wrong releasing.

Full Changelog: v0.4.1...v0.4.2

v0.4.1

🐞 Bug Fixes

• update (minimum) pydantic version to support pydantic.HttpUrl.encoded_string

Full Changelog: v0.4.0...v0.4.1

v0.4.0

What's Changed

• Improve code quality by introducing/optimising coverage reporting, typechecking, formatting, linting etc. by @adi-wan-askui in #46
• Fix ToolError by @mlikasam-askui in #47
• ci: enable linting in ci by @adi-wan-askui in #48
• feat(agent): add repeat parameter to VisionAgent.keyboard() by @adi-wan-askui in #50
• Fix routing & blocking telemetry by @adi-wan-askui in #49
• fix(agent): model composition is not supported by get/act by @adi-wan-askui in #51

🚀 Features

• repeat parameter added to VisionAgent.keyboard() for repeating key presses/releases
• count parameter added to AgentOs.keyboard_tap() and AskUiControllerClient.keyboard_tap() for repeating key presses/releases

🐞 Bug Fixes

• fix Anthropic computer use agent's handling of invalid tools and (not yet implemented) "cursor_position" tool
• fix ModelComposition not being supported as model by VisionAgent.act() and VisionAgent.get()
• raise ModelNotFoundError when model specified cannot be found instead of indicating credentials are missing
• raise ValidationError when credentials, e.g., ANTHROPIC_API_KEY or ASKUI_WORKSPACE_ID, are not set for model specified (using model parameter, e.g., of VisionAgent.click()) instead of routing to different (unexpected) model where credentials are set
• raise ValidationError when credentials, e.g., ANTHROPIC_API_KEY or ASKUI_WORKSPACE_ID, are not set for default model (when model parameter, e.g., of VisionAgent.click() is not set) instead of indicating model is invalid
• use url specified in SegmentSettings for Segment telemetry processor instead of default url
• prevent unreachable Segment url from blocking test or program termination with endless (or at least a lot of) retries ending program with a lot of errors
• fix askui.models.askui.ai_element_utils.Rectangle (xmax was falsely named ymax)
• fix askui.models.askui.ai_element_utils.AiElement.json_path
  (was falsely named metadata_path)
• fix authorization in askui.tools.askui.askui_hub.AskUIHub
• add reporting for VisionAgent.keyboard()

Other

• validate askui.models.askui.ai_element_utils.AiElement when
  deserializing from json
• introduce debugger, linting, formatting and typechecking to increase code quality
• make askui.tools.askui.askui_controller more robust, e.g., by checking if stub has been initialized before using it
• Introduce exceptions ModelNotFoundError, QueryNoResponseError, QueryUnexpectedResponseError, AskUiApiError, AskUiApiRequestFailedError, AskUiControllerError, AskUiControllerOperationFailedError, AskUiControllerOperationTimeoutError
• make model clients, e.g., ClaudeHandler, AskUiInferenceApi, etc.
  more configurable moving their settings into BaseSettings subclass
  and exposing them through constructor
• track VisionAgent.locate()
• fix telemetry disabling in tests
• enable resetting telemetry processors

🚨 Breaking Changes

• require prompt parameter of askui.locators.Prompt() to have at
  least 1 character

Full Changelog: v0.3.3...v0.4.0

v0.3.3

🐞 Bug Fixes

• exclude non-serializable parameters from telemetry

Full Changelog: v0.3.2...v0.3.3

v0.3.2

🐞 Bug Fixes

• fix serialization of Text() locator without content for AskUI models

Other

• document and export (custom) errors from exceptions modules so that they can be imported and caught

Full Changelog: v0.3.1...v0.3.2

v0.3.1

What's Changed

• Fix UI-Tars execute hotkey act by @mlikasam-askui in #43
• Optimise in-line documentation for documentation generation by @adi-wan-askui in #45

🐞 Bug Fixes

• fix UI-Tars' use of hotkey actions
• fix AskUI controller being killed instead of terminated on AskUiControllerServer.stop(force=False) or AskUiControllerServer.stop() and being terminated regularly on using AskUiControllerServer.stop(force=True)

📜 Docs

• document all public functions, classes, types etc.
• make documentation consistent
• fix obsolete references in README

Other

• expose all types (hints) for parameters and return types of public functions so that users can use them within their code

🚨 Breaking Changes

• properties of locators made private (prefixed with _, e.g., Prompt._prompt instead of Prompt.prompt) to make locator api simpler to use as properties are usually not accessed

Full Changelog: v0.3.0...v0.3.1

v0.3.0

What's Changed

• Optimize docs, naming, UX etc. by @adi-wan-askui in #42
• Model Composition / Selection by @adi-wan-askui in #41
• Get command with "askui" model (including response schema) by @adi-wan-askui in #40
• Add locators by @adi-wan-askui in #36

🚀 Features

• locate ui element using images, prompts, element classes (e.g., "textfield"), relations among ui elements
• use json schema to extract more complex data (other than strings)
• new reporters + inject your own custom reporter
• more flexible model selection + stable ocr model per default (inject through model parameter on initialization of VisionAgent or on call of method, e.g., VisionAgent.click(), as str or ModelComposition)

🐞 Bug Fixes

• fix reports overriding each other (names not unique enough)

Other

• improve and add documentation
• start AskUiControllerServer not on initialization but lazily on entering the VisionAgent context
• improve configurability/testability by allowing injection of ModelRouter, AgentToolbox and Reporter into VisionAgent on initialization (replaces enable_report, enable_askui_controller parameters)
• improve input parameter validation by validating arguments of all public function calls
• add VisionAgent.locate() method for locating (returning center position of) ui elements

🚨 Breaking Changes

• model_name parameter (e.g., VisionAgent.click() or VisionAgent.mouse_move()) renamed to model
• model value "claude" for VisionAgent.act() changed to
  "anthropic-claude-3-5-sonnet-20241022"
• change of order of parameters of VisionAgent.get()
• removed parameters of VisionAgent() (VisionAgent.__init__()): enable_report, enable_askui_controller
• "askui" model only chosen as default model if ASKUI_WORKSPACE_ID and ASKUI_TOKEN environment variables are set
• rename instruction parameter to locator (e.g., VisionAgent.click()) or query (VisionAgent.get())
• remove PC_AND_MODIFIER_KEY type in favor of PcKey | ModifierKey

Full Changelog: v0.2.5...v0.3.0

v0.2.5

What's Changed

• Chat (Experimental) by @adi-wan-askui in #33
• CL-1312 Fix Installer Download Urls by @mlikasam-askui in #35
• Setup telemetry by @adi-wan-askui in #34

New Contributors

• @mlikasam-askui made their first contribution in #35

Full Changelog: v0.2.4...v0.2.5

v0.2.4

Merge pull request #32 from askui/mouse_scroll_not_working

fix: mouse scroll not working