diff --git a/README.md b/README.md index 94e245a..7427f00 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,8 @@ [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/softpudding/OpenBrowser) +[中文 README](README.zh-CN.md) + OpenBrowser is a multimodal browser agent for real web tasks. It treats browser automation as a visual and interactive systems problem, not just a DOM parsing problem. Browsers are among the most complex pieces of software most people use every day. Reading the DOM can help, but understanding the DOM is not the same thing as actually operating the page. The long-term direction we believe in is multimodal control, or at least a strongly hybrid approach. @@ -17,24 +19,32 @@ OpenBrowser is built around that view: ## Demo -### Apartment Hunting on Xiaohongshu +### Apartment Hunting on Zillow -This demo is a better representation of what OpenBrowser is trying to do than a benchmark replay. The agent searches Xiaohongshu for apartments near Xixi Wetland, inspects multiple posts, judges renovation quality from images, likes and saves strong candidates, leaves comments, and then produces a shortlist. +This demo is a better representation of what OpenBrowser is trying to do than a benchmark replay. The agent searches Zillow for one-bedroom rentals in Capitol Hill, Seattle, opens and compares multiple listings, judges brightness, cleanliness, practicality, and value from the listing photos, and then produces a shortlist. Task prompt: -> Help me find 3 whole one-bedroom rentals near Xixi Wetland on Xiaohongshu. They should be close to a metro station, not feel too old, dark, cluttered, or overly styled, and the kitchen, bathroom, and bedroom should look clean with decent natural light. Browse multiple posts, pick the best 3, like and save them, comment on the best 2 to ask about price, earliest move-in date, short-term rental, and whether cats are allowed, then summarize why you chose them. +> Find the best 3 one-bedroom apartment rentals in Capitol Hill, Seattle on Zillow. +> +> Prioritize places that look bright, clean, practical, and close to everyday city life. +> Avoid units that look dark, cramped, outdated, cluttered, or overpriced for what they offer. +> +> Browse multiple listings (view at least 10, for better candidates), compare them visually, and return the best 3 choices with: +> 1. a one-sentence reason, +> 2. the rent, +> 3. the listing link. -![Xiaohongshu Apartment Hunting Demo](demo/xiaohongshu_apartment_preview.gif) +![Zillow Apartment Hunting Demo](demo/recording_zillow_preview.gif) -[Watch full video: xiaohongshu_3_apartments_3x.mp4](demo/xiaohongshu_3_apartments_3x.mp4) +[Watch full video: recording_zillow.webm](demo/recording_zillow.webm) What this demo shows: -- Visual judgment, not just text extraction: lighting, clutter, decoration quality, and room condition -- Real browser-side interaction: search, open posts, like, save, and comment -- Multi-step decision making across multiple candidates -- End-to-end output instead of isolated single-page actions +- Visual judgment, not just text extraction: lighting, cleanliness, layout practicality, and overall value +- Real browser-side interaction: search, open listings, compare candidates, and inspect details +- Multi-step decision making across a larger candidate set +- End-to-end output with reasons, rents, and listing links ## Why OpenBrowser diff --git a/README.zh-CN.md b/README.zh-CN.md new file mode 100644 index 0000000..fc92ee8 --- /dev/null +++ b/README.zh-CN.md @@ -0,0 +1,339 @@ +# OpenBrowser + +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/softpudding/OpenBrowser) + +[English README](README.md) + +OpenBrowser 是一个面向真实网页任务的多模态浏览器 Agent。 + +它把浏览器自动化视为一个视觉与交互系统问题，而不只是 DOM 解析问题。浏览器是大多数人每天都会使用的最复杂的软件环境之一。读取 DOM 当然有帮助，但理解 DOM 并不等于真正能把网页操作好。我们相信更长期的方向是多模态控制，或者至少是强混合式控制。 + +OpenBrowser 围绕这个判断来构建： + +- 通过截图和直接浏览器操作，以视觉方式理解和控制页面 +- 将浏览器执行路径与控制窗口隔离 +- 在 mock 网站和真实工作流上持续评测 +- 把模型成本当作一等工程约束 + +> 注意：OpenBrowser 目前仅通过 Chrome 扩展支持 Chrome 浏览器。开发和评测主要基于 `dashscope/qwen3.5-plus` 和 `dashscope/qwen3.5-flash`。 + +## Demo + +### 小红书租房 Demo + +这个 demo 更接近 OpenBrowser 想解决的真实网页任务：它会在小红书上搜索西溪湿地附近的一居室整租房源，浏览多条帖子，基于采光、整洁度、装修状态和空间实用性做视觉判断，并筛出更合适的候选。 + +任务 prompt： + +> 帮我在小红书上找 3 个西溪湿地附近的整租一居室。它们最好靠近地铁，不要太老、太暗、太乱或者过度装修；厨房、卫生间和卧室看起来要干净，并且有不错的自然采光。浏览多条帖子，选出最好的 3 个；给最好的 2 个点赞和收藏，并评论询问价格、最早入住时间、能否短租、以及是否允许养猫；最后总结你为什么选它们。 + +![小红书租房 Demo](demo/recording_xiaohongshu_preview.gif) + +[完整视频：recording_xiaohongshu.webm](demo/recording_xiaohongshu.webm) + +这个 demo 展示了： + +- 在真实中文内容平台里搜索、开帖、翻图和比较房源 +- 基于图片里的采光、整洁度、布局和装修状态做视觉判断 +- 在长流程里穿插点赞、收藏、评论等站内交互 +- 最终输出候选 shortlist，而不是停在单步按钮操作 + +## 为什么是 OpenBrowser + +### 浏览器很难 + +浏览器本身已经是工业界最复杂的软件环境之一：动态布局、异步状态、弹窗、标签页切换、可滚动容器、局部渲染，以及充满噪声的视觉上下文，都会在日常任务里同时出现。 + +### 最原生的界面是视觉 + +人类操作浏览器，本来就是靠看页面，再配合鼠标和键盘。当前模型想稳定做到这一点仍然需要大量工程辅助，但最自然的控制闭环仍然是视觉的。这也是 OpenBrowser 把截图和交互原语放在核心位置的原因。 + +### DOM 有帮助，但 DOM-only 不是终局 + +像 PinchTab 或 OpenClaw Browser Relay 这样的重 DOM 系统在今天可以工作得很好，而且在某些任务上可能比多模态流水线更快、更准。但理解 DOM 并不等于能稳定操作真实页面。我们的判断是，长期最好的浏览器 Agent 会是多模态的，或者至少是强混合式的。 + +### 评测是开发的一部分 + +OpenBrowser 不是靠“感觉不错”来迭代的。仓库里包含带事件跟踪的 mock 网站，放在 [`eval/`](eval/) 下；有意义的改动都会在这套评测上验证。真实世界里失败过的行为，也会反过来变成新的评测用例。 + +### 成本同样重要 + +模型能力重要，但价格也同样重要。我们不假设 token 成本会一直便宜下去。OpenBrowser 从一开始就把这个约束纳入设计，包括对更强模型和更便宜模型的分层处理。 + +## 评测 + +OpenBrowser 目前用两种互补方式做评测： + +- 真实浏览器工作流，以及和现有方案的并排对比 +- 带事件跟踪的自定义 mock 网站回归套件，位于 [`eval/`](eval/) + +仓库里当前主要归档对比，保持相同控制设置，对比的是 `OpenClaw Browser Relay` 和 `OpenClaw + OpenBrowser skill`： + +- [`eval/archived/2026-03-16/browser_agent_evaluation_2026-03-16_openclaw_vs_openbrowser.md`](eval/archived/2026-03-16/browser_agent_evaluation_2026-03-16_openclaw_vs_openbrowser.md) +- [`eval/evaluation_report.json`](eval/evaluation_report.json) + +我们重点跟踪： + +- 通过率 +- 执行时间 +- 成本 +- 控制窗口剩余上下文空间 + +`2026-03-16` 的代表性归档结果： + +| 方案 | 通过率 | 平均时间 | 控制窗口上下文 | +|------|--------|----------|----------------| +| OpenClaw Browser Relay | 6/7 | 211s | 640% | +| OpenClaw + OpenBrowser (`qwen3.5-plus`) | 7/7 | 274s | 21% | +| OpenClaw + OpenBrowser (`qwen3.5-flash`) | 首轮 5/7，重试后 7/7 | 317s | 12% | + +这个对比不是为了宣称 OpenBrowser 在所有任务、所有指标上都更优，而是为了把真实权衡说清楚：重 DOM relay 系统在今天可能依然很强，而 OpenBrowser 的设计目标，是保留控制窗口上下文空间，支持多模态执行路径，并通过可重复评测持续改进。 + +### 自己运行评测 + +```bash +# 列出可用测试 +python eval/evaluate_browser_agent.py --list + +# 一次性设置浏览器 capability token +export OPENBROWSER_CHROME_UUID=YOUR_BROWSER_UUID + +# 用两个模型跑全部测试 +python eval/evaluate_browser_agent.py --model dashscope/qwen3.5-plus --model dashscope/qwen3.5-flash + +# 或者在单次运行里显式传 browser UUID +python eval/evaluate_browser_agent.py --test techforum --chrome-uuid YOUR_BROWSER_UUID +``` + +评测框架说明见 [AGENTS.md](AGENTS.md#evaluation-system)。 + +## 快速开始 + +### 用你的浏览器体验 OpenBrowser + +#### 1. 安装 Python 依赖 + +```bash +# 使用 uv（推荐） +uv sync + +# 或者使用 pip +pip install -e . + +# 开发环境（包含 pytest、black、ruff 等开发依赖） +uv sync --group dev +# 或者用 pip +pip install -e ".[dev]" +``` + +#### 2. 启动服务端 + +```bash +uv run local-chrome-server serve +``` + +服务会启动在 `http://127.0.0.1:8765`（HTTP）和 `ws://127.0.0.1:8766`（WebSocket）。 + +#### 3. 配置 LLM 设置 + +第一次访问时，网页界面会提示你配置 LLM： + +1. 在浏览器中打开 `http://localhost:8765` +2. 你会看到 **Configuration Page** +3. 填写 API 配置： + - **Model**：默认是 `dashscope/qwen3.5-plus`（也支持更便宜的 `dashscope/qwen3.5-flash`） + - **Base URL**：默认是 `https://dashscope.aliyuncs.com/compatible-mode/v1` + - **API Key**：你的 API key（必填） +4. 可以按需配置 **Default Working Directory**（CWD） +5. 点击 **Save**，再点击 **Continue to Main Interface** + +> **注意**： +> - 配置会保存在 `~/.openbrowser/llm_config.json` +> - 你可以随时通过状态栏里的 **⚙️ Settings** 按钮修改设置 +> - 环境变量（LLM_API_KEY、LLM_MODEL、LLM_BASE_URL）**不再支持**，请使用 Web UI 配置 + +#### 4. 构建 Chrome 扩展 + +```bash +cd extension +npm install +npm run build +``` + +#### 5. 在 Chrome 里安装扩展 + +1. 打开 Chrome，进入 `chrome://extensions/` +2. 开启 **Developer mode**（右上角开关） +3. 点击 **Load unpacked** +4. 选择 `extension/dist` 目录 + +安装完成后，OpenBrowser 会打开一个浏览器内部页面，显示当前浏览器实例的 UUID。 +这个 UUID 就是控制该浏览器实例的权限 key。 + +重要： + +- 任何拿到这个 UUID 的人，都可以通过 OpenBrowser 操作该浏览器 +- 不要随意分享 +- 之后点击扩展图标，也可以重新打开 UUID 页面 + +#### 6. 配置 Chrome 弹窗设置（重要） + +Chrome 默认会拦截弹出窗口，这可能导致 OpenBrowser 点击链接时无法打开新标签页。你需要允许弹窗： + +**方案 A：对特定网站放行（推荐）** + +1. 当弹窗被拦截时，地址栏会显示一个被拦截图标（🚫） +2. 点击图标，选择 “Always allow pop-ups and redirects from [site]” +3. 点击 **Done** + +**方案 B：全局允许弹窗** + +1. 打开 Chrome 设置：`chrome://settings/content/popups` +2. 在 “Default behavior” 下选择 **Sites can send pop-ups and use redirects** +3. 或者把特定网站加到 “Allowed to send pop-ups” 列表里 + +> **注意**：如果 OpenBrowser 点击了链接却没有打开新标签页，先检查地址栏是否有弹窗拦截图标。这是新用户最常见的问题之一。 + +#### 7. 打开 Web 前端 + +在浏览器中访问： + +```text +http://localhost:8765 +``` + +现在你就可以通过 Web 界面和 Agent 交互了。 + +在发送指令前： + +1. 从扩展页面复制浏览器 UUID +2. 粘贴到前端的 `BROWSER UUID` 输入框 +3. 开始聊天 + +权限流转过程如下： + +1. Chrome 扩展通过 WebSocket 连接到服务端 +2. 服务端为该浏览器保存一条 `uuid -> websocket` 映射 +3. 前端会话向用户索要这个 UUID +4. 用户发送消息时，前端会把这个 UUID 一起带上 +5. 服务端据此把浏览器命令路由到对应扩展连接 + +这意味着：只要持有这个 UUID capability token，就拥有控制该浏览器的权限。 + +--- + +### 也可以通过 SKILL 使用 OpenBrowser + +直接告诉你的 agent 安装 `skill/codex/open-browser` + +## 为什么当前主要使用 Qwen3.5 系列？ + +OpenBrowser 目前主要围绕 Qwen3.5 系列开发，因为它在多模态浏览器任务上，给出了一个比较实用的能力/成本平衡点。 + +在实际使用里： + +- `qwen3.5-plus` 主要用于更难的视觉推理和更复杂的多步执行 +- `qwen3.5-flash` 更适合追求更快迭代速度和更低成本的场景 +- 这个项目把模型选择视为工程权衡，而不是产品本身 + +进一步了解 Qwen3.5： + +- [Qwen3.5: Towards Native Multimodal Agents（官方博客）](https://qwen.ai/blog/qwen3.5) +- [Qwen3.5: Towards Native Multimodal Agents（阿里云）](https://www.alibabacloud.com/blog/qwen3.5-towards-native-multimodal-agents) +- [Alibaba unveils Qwen3.5 as China's chatbot race shifts to AI agents (CNBC)](https://www.cnbc.com/2026/02/17/china-alibaba-qwen3.5-ai-agent.html) +- [Alibaba unveils new Qwen3.5 model for 'agentic AI era' (Reuters)](https://www.reuters.com/technology/alibaba-unveils-qwen3.5-agentic-ai) +- [QwenLM/Qwen3.5 (GitHub)](https://github.com/QwenLM/Qwen3.5) + +## 设计原则 + +### 1. 多模态优先，必要时混合 + +OpenBrowser 以视觉页面理解和直接交互为核心。像 DOM 这样的结构化信号依然可能有帮助，但不会被假设为完整答案。 + +### 2. 保持执行路径隔离 + +浏览器 worker 不应该把全部状态都灌进控制窗口。OpenBrowser 使用独立执行路径，让控制模型不必背负完整的浏览器会话历史。 + +### 3. 持续评测 + +仓库里包含 mock 网站、事件跟踪和归档对比结果。目标不是只把 demo 做好一次，而是在回归压力下持续变强。 + +### 4. 尊重成本约束 + +浏览器 Agent 只有在实际可运行时才有价值。因此 OpenBrowser 把价格和上下文使用量都视为核心设计约束，而不是事后补充。 + +## 核心特性 + +- **视觉 AI 自动化**：通过 AI 驱动的视觉识别来“看见”并操作网页 +- **浏览器控制**：基于视觉理解和 JavaScript 执行完成点击、输入、滚动和导航 +- **标签页管理**：支持打开、关闭、切换和管理标签页，并保持会话隔离 +- **数据提取**：利用 AI 对页面结构的理解抓取和收集网站数据 +- **表单填写与提交**：自动填写表单、提交数据，并处理多步工作流 +- **会话持久化**：在自动化任务之间保留浏览器会话、Cookie 和登录状态 +- **多接口访问**：提供 REST API、WebSocket 和 CLI，方便程序化控制 + +## 架构 + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Qwen3.5 Family (Multimodal LLM) │ +│ Qwen3.5-Plus (primary) / Qwen3.5-Flash (cost-effective) +│ Visual Perception │ Decision Making │ Browser Control │ +└─────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ OpenBrowser Agent Server (FastAPI) │ +│ REST API │ WebSocket │ Session Management │ Tool Orchestration +└─────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Chrome Extension (Chrome DevTools) │ +│ Screenshots │ JavaScript Execution │ Tab Management │ +└─────────────────────────────────────────────────────────────┘ +``` + +## 开发 + +### 构建命令 + +```bash +# 扩展开发模式构建（watch） +cd extension +npm run dev + +# TypeScript 类型检查 +npm run typecheck +``` + +### 项目结构 + +```text +. +├── server/ # FastAPI 服务端与 agent 逻辑 +│ ├── agent/ # Agent 编排 +│ ├── api/ # REST 接口 +│ ├── core/ # 核心处理逻辑 +│ └── websocket/ # WebSocket 服务 +├── extension/ # Chrome 扩展（TypeScript） +│ ├── src/ +│ │ ├── background/ # 带 CDP 的后台脚本 +│ │ ├── commands/ # 浏览器自动化命令 +│ │ └── content/ # 提供视觉反馈的内容脚本 +│ └── dist/ # 构建后的扩展 +└── frontend/ # Web UI +``` + +## 许可证 + +LGPL-3.0 + +## 致谢 + +本项目构建在 [OpenHands SDK](https://github.com/OpenHands/software-agent-sdk) 之上，它为我们的 agent 架构和工具集成提供了基础。感谢 OpenHands 团队对开源社区的贡献。 + +特别感谢： + +- **OpenHands Team**：提供了优秀的 SDK，支撑了整个 agent 系统 +- **Qwen Team (Alibaba)**：提供了强大的 Qwen3.5-Plus 多模态模型 diff --git a/demo/recording_xiaohongshu.webm b/demo/recording_xiaohongshu.webm new file mode 100644 index 0000000..1da9916 Binary files /dev/null and b/demo/recording_xiaohongshu.webm differ diff --git a/demo/recording_xiaohongshu_preview.gif b/demo/recording_xiaohongshu_preview.gif new file mode 100644 index 0000000..42330cc Binary files /dev/null and b/demo/recording_xiaohongshu_preview.gif differ diff --git a/demo/xiaohongshu_3_apartments_3x.mp4 b/demo/recording_zillow.webm similarity index 54% rename from demo/xiaohongshu_3_apartments_3x.mp4 rename to demo/recording_zillow.webm index 3963ea3..5ffc69b 100644 Binary files a/demo/xiaohongshu_3_apartments_3x.mp4 and b/demo/recording_zillow.webm differ diff --git a/demo/recording_zillow_preview.gif b/demo/recording_zillow_preview.gif new file mode 100644 index 0000000..6d35f62 Binary files /dev/null and b/demo/recording_zillow_preview.gif differ diff --git a/demo/xiaohongshu_apartment_preview.gif b/demo/xiaohongshu_apartment_preview.gif deleted file mode 100644 index 4f051b4..0000000 Binary files a/demo/xiaohongshu_apartment_preview.gif and /dev/null differ diff --git a/extension/src/commands/__tests__/tab-manager-url-normalization.test.ts b/extension/src/commands/__tests__/tab-manager-url-normalization.test.ts new file mode 100644 index 0000000..d766270 --- /dev/null +++ b/extension/src/commands/__tests__/tab-manager-url-normalization.test.ts @@ -0,0 +1,21 @@ +import { describe, expect, test } from 'bun:test'; + +import { normalizeNavigableUrl } from '../tab-manager'; + +describe('tab URL normalization', () => { + test('adds https for bare hostnames', () => { + expect(normalizeNavigableUrl('example.com')).toBe('https://example.com'); + }); + + test('preserves explicit file URLs', () => { + expect(normalizeNavigableUrl('file:///Users/test/index.html')).toBe( + 'file:///Users/test/index.html', + ); + }); + + test('preserves explicit http URLs', () => { + expect(normalizeNavigableUrl('http://localhost:8080')).toBe( + 'http://localhost:8080', + ); + }); +}); diff --git a/extension/src/commands/tab-manager.ts b/extension/src/commands/tab-manager.ts index 9bd7207..37e8b3b 100644 --- a/extension/src/commands/tab-manager.ts +++ b/extension/src/commands/tab-manager.ts @@ -8,6 +8,14 @@ const TAB_GROUP_NAME = 'OpenBrowser'; const TAB_GROUP_COLOR = 'grey' as chrome.tabGroups.Color; const TAB_GROUP_COLLAPSED = false; +const URL_WITH_SCHEME_REGEX = /^[a-zA-Z][a-zA-Z\d+.-]*:\/\//; + +export function normalizeNavigableUrl(url: string): string { + const trimmedUrl = url.trim(); + return URL_WITH_SCHEME_REGEX.test(trimmedUrl) + ? trimmedUrl + : `https://${trimmedUrl}`; +} export interface ManagedTab { tabId: number; @@ -223,13 +231,7 @@ export class TabManager { const session = this.getOrCreateSession(conversationId); - // Ensure URL has protocol - let targetUrl = startUrl; - - // FIXME(softpudding): I removed this so file can be accessed directly. - // if (!startUrl.match(/^https?:\/\//)) { - // targetUrl = `https://${startUrl}`; - // } + const targetUrl = normalizeNavigableUrl(startUrl); // First, ensure we have a tab group (find existing or create new) @@ -377,11 +379,7 @@ export class TabManager { const session = this.getOrCreateSession(conversationId); - // Ensure URL has protocol - let targetUrl = url; - if (!url.match(/^https?:\/\//)) { - targetUrl = `https://${url}`; - } + const targetUrl = normalizeNavigableUrl(url); // Create the tab const tab = await chrome.tabs.create({ url: targetUrl, active }); diff --git a/extension/src/commands/tabs.ts b/extension/src/commands/tabs.ts index 2091194..2da3bbc 100644 --- a/extension/src/commands/tabs.ts +++ b/extension/src/commands/tabs.ts @@ -2,7 +2,7 @@ * Tab Management Tool */ -import { tabManager } from './tab-manager'; +import { normalizeNavigableUrl, tabManager } from './tab-manager'; /** * Get all tabs across all windows @@ -90,11 +90,7 @@ export async function openTab( url: string, conversationId?: string, ): Promise { - // Ensure URL has protocol - let targetUrl = url; - // if (!url.match(/^https?:\/\//)) { - // targetUrl = `https://${url}`; - // } + const targetUrl = normalizeNavigableUrl(url); // Use tab manager to open managed tab if conversationId provided if (conversationId) { diff --git a/frontend/index.html b/frontend/index.html index 9af5c57..4bb3033 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -2575,13 +2575,25 @@ position: absolute; inset: 0 10px 0 0; z-index: 6; + min-height: 0; + overflow-y: auto; + overflow-x: hidden; + overscroll-behavior: contain; + scrollbar-gutter: stable; + } + + body.advanced-mode .advanced-shell summary { + position: sticky; + top: 0; + z-index: 1; + background: linear-gradient(180deg, rgba(34, 27, 21, 0.98), rgba(24, 20, 16, 0.94)); } body.advanced-mode .advanced-shell[open] .advanced-content { - flex: 1 1 auto; + flex: 1 0 auto; min-height: 0; max-height: none; - overflow-y: auto; + overflow: visible; padding-right: 6px; } @@ -3982,6 +3994,7 @@

Sisyphus

clearPendingHelpRequest(); currentConversationId = null; sisyphusCurrentConversationId = null; + resetUsageMetrics(); updateConversationIdDisplay(); updateSessionDisplay(); addEvent({ @@ -4772,56 +4785,64 @@

Sisyphus

try { const response = await fetch(apiUrl(`/agent/conversations/${conversationId}/events`)); const data = await response.json(); - - if (data.success && data.events && data.events.length > 0) { - // Clear existing events (except system init events) - const terminalOutput = document.getElementById('terminal-output'); - // Keep only the first system init message - while (terminalOutput.children.length > 1) { - terminalOutput.removeChild(terminalOutput.lastChild); + + // Clear existing events (except system init events) + const terminalOutput = document.getElementById('terminal-output'); + // Keep only the first system init message + while (terminalOutput.children.length > 1) { + terminalOutput.removeChild(terminalOutput.lastChild); + } + resetEventCounters(); + resetRecentActivity(); + resetAgentVoice(); + clearPendingHelpRequest(); + resetUsageMetrics(); + + if (!(data.success && Array.isArray(data.events))) { + return; + } + + let unresolvedHelpRequest = null; + + // Replay events + for (const event of data.events) { + const eventData = event.event_data || {}; + + if (event.event_type === 'usage_metrics') { + updateUsageMetrics(eventData); + continue; } - resetEventCounters(); - resetRecentActivity(); - resetAgentVoice(); - clearPendingHelpRequest(); - let unresolvedHelpRequest = null; - - // Replay events - for (const event of data.events) { - const eventData = event.event_data; - - // Skip SystemPromptEvent (don't show full system prompt) - if (eventData.type === 'SystemPromptEvent') { - continue; - } - - // Convert to frontend event format - const frontendEvent = normalizeFrontendEvent(eventData, { - type: event.event_type, - timestamp: event.created_at - }); - - updateEventCounters(frontendEvent.type); - addEvent(frontendEvent); - - if (frontendEvent.type === 'MessageEvent' && frontendEvent.source === 'user') { - unresolvedHelpRequest = null; - } else if (isHelpRequestEvent(frontendEvent)) { - unresolvedHelpRequest = { - conversationId: conversationId, - toolCallId: frontendEvent.tool_call_id, - message: getHelpRequestText(frontendEvent) - }; - } + // Skip SystemPromptEvent (don't show full system prompt) + if (eventData.type === 'SystemPromptEvent') { + continue; } - if (unresolvedHelpRequest) { - showHelpRequestModal(unresolvedHelpRequest, { silent: true }); + // Convert to frontend event format + const frontendEvent = normalizeFrontendEvent(eventData, { + type: event.event_type, + timestamp: event.created_at + }); + + updateEventCounters(frontendEvent.type); + addEvent(frontendEvent); + + if (frontendEvent.type === 'MessageEvent' && frontendEvent.source === 'user') { + unresolvedHelpRequest = null; + } else if (isHelpRequestEvent(frontendEvent)) { + unresolvedHelpRequest = { + conversationId: conversationId, + toolCallId: frontendEvent.tool_call_id, + message: getHelpRequestText(frontendEvent) + }; } - - console.log(`Loaded ${data.events.length} events for session ${conversationId}`); } + + if (unresolvedHelpRequest) { + showHelpRequestModal(unresolvedHelpRequest, { silent: true }); + } + + console.log(`Loaded ${data.events.length} events for session ${conversationId}`); } catch (error) { console.error('Error loading session events:', error); } @@ -4983,6 +5004,7 @@

Sisyphus

clearPendingHelpRequest(); currentConversationId = null; localStorage.removeItem('openbrowser_conversation_id'); + resetUsageMetrics(); updateConversationIdDisplay(); // Optional: Show notification @@ -5139,6 +5161,7 @@

Sisyphus

let conversationId = currentConversationId; if (!conversationId) { try { + resetUsageMetrics(); // Get cwd from input field (already validated) const response = await fetch(apiUrl('agent/conversations'), { @@ -5870,6 +5893,7 @@

Sisyphus

clearPendingHelpRequest(); currentConversationId = null; localStorage.removeItem('openbrowser_conversation_id'); + resetUsageMetrics(); updateConversationIdDisplay(); updateSessionDisplay(); @@ -6337,6 +6361,16 @@

Sisyphus

} // Update usage metrics display + function resetUsageMetrics() { + document.getElementById('usage-cost').textContent = '$0.00'; + document.getElementById('usage-prompt-tokens').textContent = '0'; + document.getElementById('usage-completion-tokens').textContent = '0'; + document.getElementById('usage-total-tokens').textContent = '0'; + document.getElementById('usage-cache-read').textContent = '0'; + document.getElementById('usage-cache-write').textContent = '0'; + document.getElementById('usage-reasoning').textContent = '0'; + } + function updateUsageMetrics(data) { console.log('[Frontend] Updating usage metrics:', data); const metrics = data.metrics || {}; diff --git a/server/agent/api.py b/server/agent/api.py index 681fb1b..4bf663e 100644 --- a/server/agent/api.py +++ b/server/agent/api.py @@ -292,6 +292,14 @@ def run_conversation(): # Put usage metrics event in queue (will be drained after complete event) if usage_metrics: logger.debug(f"DEBUG: Putting usage_metrics event into queue") + session_manager.save_event( + conversation_id=conversation_id, + event_type="usage_metrics", + event_data={ + "conversation_id": conversation_id, + "metrics": usage_metrics, + }, + ) event_queue.put( SSEEvent( "usage_metrics", diff --git a/server/core/browser_executor_bundle.py b/server/core/browser_executor_bundle.py index da1b78b..6c0613d 100644 --- a/server/core/browser_executor_bundle.py +++ b/server/core/browser_executor_bundle.py @@ -21,6 +21,7 @@ from server.api.sse import SSEEvent from server.agent.user_help import build_completion_event_payload from server.core.processor import CommandProcessor +from server.core.session_manager import session_manager from server.models.commands import parse_command, CommandResponse logger = logging.getLogger(__name__) @@ -309,6 +310,14 @@ async def execute_agent_message( ) if usage_metrics: + session_manager.save_event( + conversation_id=self.conversation_id, + event_type="usage_metrics", + event_data={ + "conversation_id": self.conversation_id, + "metrics": usage_metrics, + }, + ) event_queue.put( SSEEvent( "usage_metrics", diff --git a/server/models/commands.py b/server/models/commands.py index 514fe38..fc13987 100644 --- a/server/models/commands.py +++ b/server/models/commands.py @@ -3,6 +3,8 @@ from pydantic import BaseModel, Field, validator, model_validator import re +URL_WITH_SCHEME_RE = re.compile(r"^[a-zA-Z][a-zA-Z0-9+.-]*://") + class MouseButton(str, Enum): LEFT = "left" @@ -157,8 +159,8 @@ class TabCommand(BaseCommand): def validate_url(cls, v, values): action = values.get("action") if action in [TabAction.OPEN, TabAction.INIT]: - # Ensure URL has protocol - if v and not re.match(r"^https?://", v): + # Add a default scheme only when the caller did not provide one. + if v and not URL_WITH_SCHEME_RE.match(v): v = f"https://{v}" return v diff --git a/server/tests/unit/test_agent_api_multiprocess.py b/server/tests/unit/test_agent_api_multiprocess.py index 91c5ddb..cb820cc 100644 --- a/server/tests/unit/test_agent_api_multiprocess.py +++ b/server/tests/unit/test_agent_api_multiprocess.py @@ -56,3 +56,63 @@ async def test_process_agent_message_uses_worker_queues() -> None: mock_session_manager.update_session_status.assert_any_call( "conv-123", SessionStatus.IDLE ) + + +@pytest.mark.asyncio +async def test_process_agent_message_persists_usage_metrics_for_history() -> None: + """Thread-mode streaming should persist usage metrics for session replay.""" + with ( + patch("server.agent.api.agent_manager") as mock_manager, + patch("server.agent.api.session_manager") as mock_session_manager, + patch( + "server.agent.api.build_completion_event_payload", + return_value={ + "conversation_id": "conv-456", + "message": "Conversation completed", + }, + ), + ): + mock_manager.multi_process_mode = False + + mock_visualizer = MagicMock() + mock_conversation = MagicMock() + mock_conversation.run = MagicMock() + mock_conversation.close = MagicMock() + mock_conversation.state.events = [] + + mock_combined_metrics = MagicMock() + mock_combined_metrics.get.return_value = {"accumulated_cost": 1.25} + mock_combined_metrics.model_name = "dashscope/qwen3.5-plus" + + mock_stats = MagicMock() + mock_stats.get_combined_metrics.return_value = mock_combined_metrics + mock_conversation.conversation_stats = mock_stats + + mock_conv_state = MagicMock( + conversation=mock_conversation, + visualizer=mock_visualizer, + ) + mock_manager.get_or_create_conversation.return_value = mock_conv_state + + sse_payloads = [ + payload + async for payload in process_agent_message( + "conv-456", + "hello thread", + cwd="/tmp/workspace", + ) + ] + + assert any("event: complete" in payload for payload in sse_payloads) + assert any("event: usage_metrics" in payload for payload in sse_payloads) + mock_session_manager.save_event.assert_called_once_with( + conversation_id="conv-456", + event_type="usage_metrics", + event_data={ + "conversation_id": "conv-456", + "metrics": { + "accumulated_cost": 1.25, + "model_name": "dashscope/qwen3.5-plus", + }, + }, + ) diff --git a/server/tests/unit/test_browser_executor_bundle.py b/server/tests/unit/test_browser_executor_bundle.py index 40b153a..b87a357 100644 --- a/server/tests/unit/test_browser_executor_bundle.py +++ b/server/tests/unit/test_browser_executor_bundle.py @@ -385,6 +385,9 @@ async def test_execute_agent_message_streams_events(self): patch( "server.core.browser_executor_bundle.CommandProcessor" ) as mock_processor_class, + patch( + "server.core.browser_executor_bundle.session_manager" + ) as mock_session_manager, ): mock_visualizer = MagicMock() mock_conversation = MagicMock() @@ -423,6 +426,17 @@ async def test_execute_agent_message_streams_events(self): assert complete_event.event_type == "complete" assert usage_event.event_type == "usage_metrics" assert usage_event.data["metrics"]["model_name"] == "test-model" + mock_session_manager.save_event.assert_called_once_with( + conversation_id="test-conv-12b", + event_type="usage_metrics", + event_data={ + "conversation_id": "test-conv-12b", + "metrics": { + "accumulated_cost": 0.5, + "model_name": "test-model", + }, + }, + ) @pytest.mark.asyncio async def test_pause_conversation_requests_pause_on_worker_conversation(self): diff --git a/server/tests/unit/test_command_models.py b/server/tests/unit/test_command_models.py index 639464c..0459383 100644 --- a/server/tests/unit/test_command_models.py +++ b/server/tests/unit/test_command_models.py @@ -22,6 +22,14 @@ def test_navigation_actions_normalize_urls(self, action: TabAction) -> None: assert command.url == "https://example.com" + @pytest.mark.parametrize("action", [TabAction.INIT, TabAction.OPEN]) + def test_navigation_actions_preserve_explicit_file_urls( + self, action: TabAction + ) -> None: + command = TabCommand(action=action, url="file:///tmp/index.html") + + assert command.url == "file:///tmp/index.html" + @pytest.mark.parametrize("action", [TabAction.INIT, TabAction.OPEN]) def test_navigation_actions_require_url(self, action: TabAction) -> None: with pytest.raises(ValidationError, match="URL is required"):