Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
4235c36
feat: Add voice assistant with dual frontend architecture
jck411 Dec 14, 2025
84274a2
WORKING WITH WEBSITE EXAMPLE STT
jck411 Dec 15, 2025
f037807
feat: Add configurable Kiosk STT settings for Deepgram EOT and keyterms.
jck411 Dec 16, 2025
76d65cf
feat: Implement configurable kiosk UI settings and display assistant …
jck411 Dec 16, 2025
82ca78a
chore: Remove stale static frontend build assets
jck411 Dec 16, 2025
4e05e4c
DEEPGRAM STT/TTS WORKING feat: Add configurable kiosk TTS settings, i…
jck411 Dec 16, 2025
b6ec695
LLM REPLIES ONCE feat: Introduce kiosk LLM settings management and in…
jck411 Dec 16, 2025
dd3b42f
conversation working, UI messaed up
jck411 Dec 16, 2025
ebbfcaf
feat: Refactor transcription screen UI with auto-scroll, updated stat…
jck411 Dec 16, 2025
5ad81b5
feat: Enable kiosk TTS
jck411 Dec 16, 2025
68fa4fc
feat: Add TTS playback event handling to synchronize global client st…
jck411 Dec 16, 2025
8287299
WORKING GREAT, UXI STILL SUX ...feat: Implement backend-driven idle …
jck411 Dec 16, 2025
46779e1
Added mcp servers for kiosk and improved settings structure. feat: Im…
jck411 Dec 16, 2025
4a1096e
KIOSK TOOLS WORKING!
jck411 Dec 16, 2025
e7f65c7
feat: Add tool event broadcasting to kiosk chat service and update da…
jck411 Dec 16, 2025
c639d44
added tools notification ok kiosk UI . feat: Update kiosk frontend wi…
jck411 Dec 16, 2025
8941944
feat: Introduce LLM configuration to Kiosk settings UI, update defaul…
jck411 Dec 17, 2025
c271dc5
feat: Implement kiosk preset management with new backend services, sc…
jck411 Dec 17, 2025
9c82493
feat: Add UI for continuous conversation mode settings and update def…
jck411 Dec 17, 2025
885d552
feat: Implement continuous conversation mode with UI updates, conditi…
jck411 Dec 17, 2025
9728169
feat: Implement streaming LLM responses with real-time tool status up…
jck411 Dec 17, 2025
33ef981
feat: Add chunked TTS audio streaming and barge-in support with STT s…
jck411 Dec 17, 2025
40391ea
feat: add ElevenLabs, OpenAI, and Unreal Speech TTS providers with up…
jck411 Dec 17, 2025
2c9f43d
feat: Add dynamic TTS sample rate handling to backend and frontend, a…
jck411 Dec 17, 2025
99533d1
feat: Implement streaming TTS with segment handling and support for O…
jck411 Dec 17, 2025
50d48a6
feat: Enhance voice assistant TTS streaming with improved sentence sp…
jck411 Dec 17, 2025
08be3b7
feat: Immediately yield first TTS audio chunk and add performance log…
jck411 Dec 17, 2025
7b9d6d0
feat: Integrate OpenAI TTS with new segmentation and detailed voice s…
jck411 Dec 18, 2025
33677a9
feat: Refactor client settings management by introducing a generic cl…
jck411 Dec 19, 2025
8c33f36
feat: Implement OpenAI streaming TTS and refactor the core TTS proces…
jck411 Dec 19, 2025
888b936
feat: Implement OpenAI TTS and refactor TTS service to remove old pro…
jck411 Dec 19, 2025
028806b
feat: update preset default status and timestamp, add new suggestions…
jck411 Dec 19, 2025
fe2cc2a
refactor: Move shell chat CLI to a dedicated `frontend-cli` package a…
jck411 Dec 19, 2025
0acb4a6
feat: add interactive bash script to launch and manage application co…
jck411 Dec 19, 2025
252551e
refactor: Consolidate preset and model settings into client settings …
jck411 Dec 19, 2025
60289ca
feat: Update default LLM, clear suggestions, and refactor MCP server …
jck411 Dec 19, 2025
1e60f13
feat: Implement client-specific MCP server configuration, update serv…
jck411 Dec 19, 2025
a1d113b
config: Update default LLM model and prompt, adjust server enablement…
jck411 Dec 19, 2025
acbb7e1
feat: Migrate MCP server enablement to a per-client `client_enabled` …
jck411 Dec 19, 2025
51d22bf
refactor: remove client_enabled field from mcp_servers.json
jck411 Dec 19, 2025
3c81f61
feat: Add CLI-specific LLM settings management with dedicated UI.
jck411 Dec 20, 2025
5706b0f
feat: add MCP servers modal and related styles; enhance speech and sy…
jck411 Dec 20, 2025
bd0cce2
feat: Integrate Google, Spotify, and Monarch Money authentication wit…
jck411 Dec 20, 2025
6086cc7
feat: Swap system settings and MCP servers button functionality in Ch…
jck411 Dec 20, 2025
d17befa
feat: Implement auto-sizing for text areas in CLI and Kiosk settings …
jck411 Dec 20, 2025
82b9328
feat: Add eager_eot_threshold to Kiosk STT settings and update relate…
jck411 Dec 20, 2025
5abe6d1
feat: Enhance client ID resolution and model settings handling in cha…
jck411 Dec 20, 2025
19dc542
feat: add system maintenance tools and update default LLM model, syst…
jck411 Dec 21, 2025
5261d44
feat: add HTTP transport support to MCP servers, introduce a new scri…
jck411 Dec 22, 2025
db0ad39
refactor: delete `run_mcp_servers.py` and update `mcp_servers.json` t…
jck411 Dec 22, 2025
bd3d897
Refactor code structure for improved readability and maintainability
jck411 Dec 22, 2025
3e768f1
feat: enable kiosk mode for multiple MCP servers, update host profile…
jck411 Dec 22, 2025
bce97bd
feat: Implement client profiles for dynamic MCP server selection and …
jck411 Dec 22, 2025
fffe5a4
feat: Implement lazy MCP server loading and discovery to improve back…
jck411 Dec 23, 2025
7b285e2
feat: Remove `/mode` command from shell_chat and add pre-start proces…
jck411 Dec 23, 2025
9619ce7
feat: Refactor server row layout to use grid for improved responsiven…
jck411 Dec 23, 2025
50f1466
feat: Add standalone MCP settings page and enable direct access via U…
jck411 Dec 23, 2025
7eb512a
refactor: Remove explicit backend client enablement logic and UI, rep…
jck411 Dec 24, 2025
29548bc
feat: update Svelte LLM configurations, enable Svelte for an MCP serv…
jck411 Dec 25, 2025
dad21a4
added inventory to to separate from profile and make leaner
jck411 Dec 25, 2025
938a7c9
added resource to prompt injection from host profile
jck411 Dec 25, 2025
2a793c4
feat: implement model filters for client presets and add a new method…
jck411 Dec 25, 2025
0eab910
feat: enhance preset handling by adding model filters and updating pr…
jck411 Dec 25, 2025
4dd9d00
feat: Refactor preset handling in API client, ensure system prompt pe…
jck411 Dec 26, 2025
4e7528e
feat: add ydotool-based UI automation tools for typing, key presses, …
jck411 Dec 26, 2025
984bce6
feat: add ui_batch tool for executing multiple UI actions in sequence
jck411 Dec 26, 2025
6d45975
feat: refactor UI action functions to improve internal implementation…
jck411 Dec 26, 2025
090ab3c
feat: Update default LLM model and system prompts to a detailed deskt…
jck411 Dec 27, 2025
9e1c47d
feat: add Playwright server for browser automation and integrate it i…
jck411 Dec 28, 2025
68344c2
feat: Introduce Playwright tool support, update LLM browser automatio…
jck411 Dec 29, 2025
b7291ac
feat: Add playwright and its greenlet dependency to the project.
jck411 Dec 29, 2025
2fae40b
feat: Add `browser_press_key` tool and update Playwright server to la…
jck411 Dec 29, 2025
9c25a3d
feat: add UI for agent invocation limit and improve playwright `type_…
jck411 Dec 29, 2025
5a1b345
feat: Add browser status tool, enable session reuse, and implement pr…
jck411 Dec 30, 2025
eeb6538
fix: Improve error handling in browser closure and session management…
jck411 Dec 30, 2025
e1c078c
feat: Enhance Playwright server with CDP port resolution and browser …
jck411 Dec 30, 2025
6941371
feat: Implement tool error handling and policy enforcement in Streami…
jck411 Dec 30, 2025
3db429d
Isolate Playwright browser profile for CDP
jck411 Jan 7, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added 20251227_16h50m55s_grim.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Empty file added =1.57.0
Empty file.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,7 @@ MCP servers are configured in `data/mcp_servers.json` and hot-reloaded via API.
- **PDF tools** — extract text and metadata
- **Monarch Money** — personal finance data and transactions
- **Calculator & utilities** — housekeeping helpers
- Local servers run over streamable HTTP using `http_port`; remote servers use `http_url`.
- The canonical list of bundled servers lives in `src/backend/mcp_servers/__init__.py`
(`BUILTIN_MCP_SERVER_DEFINITIONS`). The FastAPI factory consumes that list to
generate default entries with the same enable/disable defaults, so updating the
Expand Down
Binary file added calendar_screenshot.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
17 changes: 17 additions & 0 deletions data/client_profiles/cli-default.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"description": "Default CLI profile with all servers enabled",
"enabled_servers": [
"custom-calendar",
"custom-gdrive",
"custom-gmail",
"custom-pdf",
"housekeeping",
"local-calculator",
"monarch-money",
"notes",
"playwright",
"shell-control",
"spotify"
],
"profile_id": "cli-default"
}
7 changes: 7 additions & 0 deletions data/client_profiles/cli-minimal.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"description": "Minimal CLI profile with only shell control",
"enabled_servers": [
"shell-control"
],
"profile_id": "cli-minimal"
}
11 changes: 11 additions & 0 deletions data/client_profiles/kiosk.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"description": "Kiosk profile for voice assistant",
"enabled_servers": [
"custom-calendar",
"custom-gmail",
"housekeeping",
"notes",
"spotify"
],
"profile_id": "kiosk"
}
16 changes: 16 additions & 0 deletions data/client_profiles/svelte.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"description": "Default Svelte frontend profile with all servers enabled",
"enabled_servers": [
"custom-calendar",
"custom-gdrive",
"custom-gmail",
"custom-pdf",
"housekeeping",
"local-calculator",
"monarch-money",
"notes",
"shell-control",
"spotify"
],
"profile_id": "svelte"
}
130 changes: 99 additions & 31 deletions data/mcp_servers.json
Original file line number Diff line number Diff line change
@@ -1,95 +1,163 @@
{
"servers": [
{
"contexts": [],
"disabled_tools": [],
"enabled": true,
"env": {
"HOST_PROFILE_ID": "xps13",
"HOST_ROOT_PATH": "/home/human/GoogleDrive/host_profiles"
"client_enabled": {
"backend": false,
"cli": true,
"kiosk": true,
"svelte": true
},
"id": "shell-control",
"module": "backend.mcp_servers.shell_control_server",
"contexts": [],
"enabled": false,
"env": {},
"http_port": 9003,
"id": "local-calculator",
"module": "backend.mcp_servers.calculator_server",
"tool_overrides": {}
},
{
"client_enabled": {
"backend": false,
"cli": true,
"kiosk": true,
"svelte": true
},
"contexts": [],
"disabled_tools": [],
"enabled": false,
"enabled": true,
"env": {},
"id": "notes",
"module": "backend.mcp_servers.notes_server",
"http_port": 9002,
"id": "housekeeping",
"module": "backend.mcp_servers.housekeeping_server",
"tool_overrides": {}
},
{
"client_enabled": {
"cli": true,
"kiosk": true,
"svelte": false
},
"contexts": [],
"enabled": false,
"env": {},
"id": "monarch-money",
"module": "backend.mcp_servers.monarch_server",
"http_port": 9004,
"id": "custom-calendar",
"module": "backend.mcp_servers.calendar_server",
"tool_overrides": {}
},
{
"client_enabled": {
"cli": true,
"kiosk": true,
"svelte": true
},
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {},
"http_port": 9005,
"id": "custom-gmail",
"module": "backend.mcp_servers.gmail_server",
"tool_overrides": {}
},
{
"client_enabled": {
"cli": true,
"kiosk": true,
"svelte": true
},
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {},
"http_port": 9006,
"id": "custom-gdrive",
"module": "backend.mcp_servers.gdrive_server",
"tool_overrides": {}
},
{
"client_enabled": {
"cli": true,
"kiosk": true,
"svelte": true
},
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {},
"http_port": 9007,
"id": "custom-pdf",
"module": "backend.mcp_servers.pdf_server",
"tool_overrides": {}
},
{
"client_enabled": {
"cli": true,
"kiosk": true,
"svelte": true
},
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {},
"id": "custom-calendar",
"module": "backend.mcp_servers.calendar_server",
"http_port": 9008,
"id": "monarch-money",
"module": "backend.mcp_servers.monarch_server",
"tool_overrides": {}
},
{
"client_enabled": {
"cli": true,
"kiosk": true,
"svelte": true
},
"contexts": [],
"disabled_tools": [],
"enabled": true,
"enabled": false,
"env": {},
"id": "spotify",
"module": "backend.mcp_servers.spotify_server",
"http_port": 9009,
"id": "notes",
"module": "backend.mcp_servers.notes_server",
"tool_overrides": {}
},
{
"client_enabled": {
"backend": true,
"cli": true,
"kiosk": true,
"svelte": true
},
"contexts": [],
"enabled": true,
"env": {
"HOST_PROFILE_ID": "xps13",
"HOST_ROOT_PATH": "/home/human/GoogleDrive/host_profiles"
},
"http_port": 9001,
"id": "shell-control",
"module": "backend.mcp_servers.shell_control_server",
"tool_overrides": {}
},
{
"client_enabled": {
"cli": true,
"kiosk": true,
"svelte": true
},
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {},
"id": "local-calculator",
"module": "backend.mcp_servers.calculator_server",
"http_port": 9010,
"id": "spotify",
"module": "backend.mcp_servers.spotify_server",
"tool_overrides": {}
},
{
"client_enabled": {
"cli": true,
"kiosk": true,
"svelte": true
},
"contexts": [],
"disabled_tools": [],
"enabled": false,
"enabled": true,
"env": {},
"id": "housekeeping",
"module": "backend.mcp_servers.housekeeping_server",
"http_port": 9011,
"id": "playwright",
"module": "backend.mcp_servers.playwright_server",
"tool_overrides": {}
}
]
Expand Down
116 changes: 116 additions & 0 deletions data/mcp_servers.json.backup
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
{
"servers": [
{
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {
"HOST_PROFILE_ID": "xps13",
"HOST_ROOT_PATH": "/home/human/GoogleDrive/host_profiles"
},
"frontend_enabled": true,
"id": "shell-control",
"kiosk_enabled": true,
"module": "backend.mcp_servers.shell_control_server",
"tool_overrides": {}
},
{
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {},
"frontend_enabled": false,
"id": "notes",
"kiosk_enabled": true,
"module": "backend.mcp_servers.notes_server",
"tool_overrides": {}
},
{
"contexts": [],
"enabled": false,
"env": {},
"frontend_enabled": true,
"id": "monarch-money",
"kiosk_enabled": false,
"module": "backend.mcp_servers.monarch_server",
"tool_overrides": {}
},
{
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {},
"frontend_enabled": true,
"id": "custom-gmail",
"kiosk_enabled": false,
"module": "backend.mcp_servers.gmail_server",
"tool_overrides": {}
},
{
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {},
"frontend_enabled": false,
"id": "custom-gdrive",
"kiosk_enabled": true,
"module": "backend.mcp_servers.gdrive_server",
"tool_overrides": {}
},
{
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {},
"frontend_enabled": true,
"id": "custom-pdf",
"kiosk_enabled": false,
"module": "backend.mcp_servers.pdf_server",
"tool_overrides": {}
},
{
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {},
"frontend_enabled": true,
"id": "custom-calendar",
"kiosk_enabled": false,
"module": "backend.mcp_servers.calendar_server",
"tool_overrides": {}
},
{
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {},
"frontend_enabled": true,
"id": "spotify",
"kiosk_enabled": false,
"module": "backend.mcp_servers.spotify_server",
"tool_overrides": {}
},
{
"contexts": [],
"disabled_tools": [],
"enabled": false,
"env": {},
"frontend_enabled": false,
"id": "local-calculator",
"kiosk_enabled": true,
"module": "backend.mcp_servers.calculator_server",
"tool_overrides": {}
},
{
"contexts": [],
"disabled_tools": [],
"enabled": true,
"env": {},
"frontend_enabled": true,
"id": "housekeeping",
"kiosk_enabled": false,
"module": "backend.mcp_servers.housekeeping_server",
"tool_overrides": {}
}
]
}
Loading