Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,16 @@ Worker vars: `ELEVENLABS_VOICE_ID`

**Transient Cursor Mode**: When "Show Clicky" is off, pressing the hotkey fades in the cursor overlay for the duration of the interaction (recording → response → TTS → optional pointing), then fades it out automatically after 1 second of inactivity.

**Manual TFT Snapshot Mode**: TFT coaching uses a local hardcoded patch/meta snapshot (`TFTMetaContext.swift`) appended to the Claude system prompt. The snapshot is intentionally manual and refreshed via `scripts/update_tft_meta_snapshot.py`, so the app stays self-contained and does not depend on live meta APIs at runtime.

## Key Files

| File | Lines | Purpose |
|------|-------|---------|
| `leanring_buddyApp.swift` | ~89 | Menu bar app entry point. Uses `@NSApplicationDelegateAdaptor` with `CompanionAppDelegate` which creates `MenuBarPanelManager` and starts `CompanionManager`. No main window — the app lives entirely in the status bar. |
| `CompanionManager.swift` | ~1026 | Central state machine. Owns dictation, shortcut monitoring, screen capture, Claude API, ElevenLabs TTS, and overlay management. Tracks voice state (idle/listening/processing/responding), conversation history, model selection, and cursor visibility. Coordinates the full push-to-talk → screenshot → Claude → TTS → pointing pipeline. |
| `CompanionManager.swift` | ~1080 | Central state machine. Owns dictation, shortcut monitoring, screen capture, Claude API, ElevenLabs TTS, and overlay management. Tracks voice state (idle/listening/processing/responding), conversation history, model/assistant mode selection, and cursor visibility. Coordinates the full push-to-talk → screenshot → Claude → TTS → pointing pipeline. |
| `MenuBarPanelManager.swift` | ~243 | NSStatusItem + custom NSPanel lifecycle. Creates the menu bar icon, manages the floating companion panel (show/hide/position), installs click-outside-to-dismiss monitor. |
| `CompanionPanelView.swift` | ~761 | SwiftUI panel content for the menu bar dropdown. Shows companion status, push-to-talk instructions, model picker (Sonnet/Opus), permissions UI, DM feedback button, and quit button. Dark aesthetic using `DS` design system. |
| `CompanionPanelView.swift` | ~830 | SwiftUI panel content for the menu bar dropdown. Shows companion status, push-to-talk instructions, assistant mode picker (General/TFT Coach), model picker (Sonnet/Opus), permissions UI, DM feedback button, and quit button. Dark aesthetic using `DS` design system. |
| `OverlayWindow.swift` | ~881 | Full-screen transparent overlay hosting the blue cursor, response text, waveform, and spinner. Handles cursor animation, element pointing with bezier arcs, multi-monitor coordinate mapping, and fade-out transitions. |
| `CompanionResponseOverlay.swift` | ~217 | SwiftUI view for the response text bubble and waveform displayed next to the cursor in the overlay. |
| `CompanionScreenCaptureUtility.swift` | ~132 | Multi-monitor screenshot capture using ScreenCaptureKit. Returns labeled image data for each connected display. |
Expand All @@ -72,9 +74,11 @@ Worker vars: `ELEVENLABS_VOICE_ID`
| `ElementLocationDetector.swift` | ~335 | Detects UI element locations in screenshots for cursor pointing. |
| `DesignSystem.swift` | ~880 | Design system tokens — colors, corner radii, shared styles. All UI references `DS.Colors`, `DS.CornerRadius`, etc. |
| `ClickyAnalytics.swift` | ~121 | PostHog analytics integration for usage tracking. |
| `TFTMetaContext.swift` | ~65 | Hardcoded TFT patch/meta snapshot and prompt context builder used by TFT Coach mode. |
| `WindowPositionManager.swift` | ~262 | Window placement logic, Screen Recording permission flow, and accessibility permission helpers. |
| `AppBundleConfiguration.swift` | ~28 | Runtime configuration reader for keys stored in the app bundle Info.plist. |
| `worker/src/index.ts` | ~142 | Cloudflare Worker proxy. Three routes: `/chat` (Claude), `/tts` (ElevenLabs), `/transcribe-token` (AssemblyAI temp token). |
| `scripts/update_tft_meta_snapshot.py` | ~185 | Manual updater script that refreshes the hardcoded TFT snapshot block from official patch notes + Data Dragon versions. |

## Build & Run

Expand Down
79 changes: 79 additions & 0 deletions docs/experiment-tft-helper-manual-snapshot.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Experiment: TFT Helper Manual Snapshot

## Summary
Added a new assistant mode called `TFT Coach` so Clicky can answer with TFT-focused guidance using a local patch/meta snapshot that is manually refreshed.

## What Changed
- Added assistant mode selection in the menu bar panel (`General` vs `TFT Coach`).
- Added a local TFT snapshot source in `TFTMetaContext.swift`.
- Appended TFT-specific instructions + snapshot context into the Claude system prompt when `TFT Coach` mode is active.
- Added panel status text so users can see which snapshot is currently loaded.
- Added analytics event for mode switching (`assistant_mode_selected`).
- Added `scripts/update_tft_meta_snapshot.py` to refresh the hardcoded snapshot when new patch notes release.
- Added tests for TFT prompt context/status builders.

## Why
The app previously had only a generic assistant prompt. Even with a TFT-looking screen, it had no stable, patch-aware TFT context and no dedicated behavior mode.

## Root Cause
- No domain mode switch existed in `CompanionManager`/`CompanionPanelView`.
- No local data source existed for current TFT patch/meta context.
- No maintenance workflow existed for patch refreshes.

## What Was Tried and Rejected
- Initial direction was a live worker endpoint for TFT meta data.
- Rejected per request to keep this hardcoded/local for now and manually updated per patch.

## Key Files
- `leanring-buddy/CompanionManager.swift`
- `leanring-buddy/CompanionPanelView.swift`
- `leanring-buddy/TFTMetaContext.swift`
- `leanring-buddy/ClickyAnalytics.swift`
- `leanring-buddyTests/leanring_buddyTests.swift`
- `scripts/update_tft_meta_snapshot.py`

## Gotchas
- Auto patch discovery from the en-us tag page may lag or differ by locale. Use `--patch-url` when needed:
- `python3 scripts/update_tft_meta_snapshot.py --patch-url <official-patch-url>`
- Snapshot is intentionally manual. If this file is stale, coach quality will drift.
- This is not live ladder ingestion; recommendations still depend on screenshot state.

## Verified Live Surface
- Exact route used: macOS menu bar panel (`NSStatusItem` -> `CompanionPanelView`), no web route.
- Files that render it:
- `leanring-buddy/CompanionPanelView.swift`
- Code path that reaches it:
- `leanring_buddyApp.swift` -> `CompanionAppDelegate` -> `MenuBarPanelManager` -> `CompanionPanelView`
- Stale routes intentionally not touched:
- No old/duplicate web routes exist (menu bar app only).

## Verification
Within 5 minutes, confirm with these checks:

1. UI mode visibility
- Launch app, open panel.
- Confirm `Mode` segmented control appears with `General` and `TFT Coach`.
- Switch to `TFT Coach`.
- Confirm the status line appears: `Manual TFT snapshot: ...`.

2. Prompt context instrumentation
- Trigger push-to-talk once in `TFT Coach`.
- Check app logs for structured prompt context event:
- grep pattern: `"event":"clicky_prompt_context"`
- expected fields include:
- `"assistantMode":"tft_coach"`
- `"transcriptCharacterCount":<number>`

3. Analytics signal
- In PostHog, verify event `assistant_mode_selected` is emitted with property `mode=tft_coach` and `mode=general`.

4. Snapshot refresh script
- Run:
- `python3 scripts/update_tft_meta_snapshot.py`
- Confirm `leanring-buddy/TFTMetaContext.swift` snapshot block updates between markers:
- `// BEGIN_AUTOGEN_TFT_SNAPSHOT`
- `// END_AUTOGEN_TFT_SNAPSHOT`

5. Unit test coverage
- Confirm tests exist for TFT prompt context/status builder in:
- `leanring-buddyTests/leanring_buddyTests.swift`
7 changes: 7 additions & 0 deletions leanring-buddy/ClickyAnalytics.swift
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,13 @@ enum ClickyAnalytics {

// MARK: - Voice Interaction

/// User switched assistant behavior mode from the panel.
static func trackAssistantModeSelected(mode: String) {
PostHogSDK.shared.capture("assistant_mode_selected", properties: [
"mode": mode
])
}

/// User pressed the push-to-talk shortcut (control+option) to start talking.
static func trackPushToTalkStarted() {
PostHogSDK.shared.capture("push_to_talk_started")
Expand Down
73 changes: 72 additions & 1 deletion leanring-buddy/CompanionManager.swift
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,20 @@ enum CompanionVoiceState {
case responding
}

enum CompanionAssistantMode: String, CaseIterable {
case general = "general"
case tftCoach = "tft_coach"

var panelLabel: String {
switch self {
case .general:
return "General"
case .tftCoach:
return "TFT Coach"
}
}
}

@MainActor
final class CompanionManager: ObservableObject {
@Published private(set) var voiceState: CompanionVoiceState = .idle
Expand Down Expand Up @@ -109,13 +123,32 @@ final class CompanionManager: ObservableObject {

/// The Claude model used for voice responses. Persisted to UserDefaults.
@Published var selectedModel: String = UserDefaults.standard.string(forKey: "selectedClaudeModel") ?? "claude-sonnet-4-6"
/// The assistant behavior mode shown in the panel.
@Published var selectedAssistantMode: CompanionAssistantMode = {
guard let storedMode = UserDefaults.standard.string(forKey: "selectedAssistantMode"),
let resolvedMode = CompanionAssistantMode(rawValue: storedMode) else {
return .general
}
return resolvedMode
}()
/// Human-readable status for the TFT snapshot shown in the panel.
@Published private(set) var tftMetaStatusMessage: String = TFTMetaPromptBuilder.buildStatusMessage()

func setSelectedModel(_ model: String) {
selectedModel = model
UserDefaults.standard.set(model, forKey: "selectedClaudeModel")
claudeAPI.model = model
}

func setSelectedAssistantMode(_ mode: CompanionAssistantMode) {
selectedAssistantMode = mode
UserDefaults.standard.set(mode.rawValue, forKey: "selectedAssistantMode")
if mode == .tftCoach {
tftMetaStatusMessage = TFTMetaPromptBuilder.buildStatusMessage()
}
ClickyAnalytics.trackAssistantModeSelected(mode: mode.rawValue)
}

/// User preference for whether the Clicky cursor should be shown.
/// When toggled off, the overlay is hidden and push-to-talk is disabled.
/// Persisted to UserDefaults so the choice survives app restarts.
Expand Down Expand Up @@ -576,6 +609,18 @@ final class CompanionManager: ObservableObject {
- element is on screen 2 (not where cursor is): "that's over on your other monitor — see the terminal window? [POINT:400,300:terminal:screen2]"
"""

/// Additional instructions when the assistant runs in TFT coach mode.
/// This is appended to the default companion prompt.
private static let tftCoachModePrompt = """
you are in teamfight tactics coach mode.

extra rules for tft coach mode:
- prioritize advice for the board and shop you currently see, then use the patch snapshot as supporting context.
- if meta guidance is uncertain, say that plainly instead of pretending confidence.
- keep recommendations actionable: what to buy, what to itemize, what to level, and which unit placement changes matter most.
- when the screenshot shows a clear next move, point directly at the relevant unit, shop slot, item, augment, or trait panel.
"""

// MARK: - AI Response Pipeline

/// Captures a screenshot, sends it along with the transcript to Claude,
Expand Down Expand Up @@ -610,9 +655,35 @@ final class CompanionManager: ObservableObject {
(userPlaceholder: entry.userTranscript, assistantResponse: entry.assistantResponse)
}

let resolvedSystemPrompt: String = {
guard selectedAssistantMode == .tftCoach else {
return Self.companionVoiceResponseSystemPrompt
}

let tftContext = TFTMetaPromptBuilder.buildPromptContext()
tftMetaStatusMessage = TFTMetaPromptBuilder.buildStatusMessage()

return Self.companionVoiceResponseSystemPrompt
+ "\n\n"
+ Self.tftCoachModePrompt
+ "\n\nmanual tft snapshot context:\n"
+ tftContext
}()

let promptLogPayload: [String: Any] = [
"event": "clicky_prompt_context",
"assistantMode": selectedAssistantMode.rawValue,
"transcriptCharacterCount": transcript.count,
"conversationHistoryCount": conversationHistory.count
]
if let logData = try? JSONSerialization.data(withJSONObject: promptLogPayload),
let logMessage = String(data: logData, encoding: .utf8) {
print(logMessage)
}

let (fullResponseText, _) = try await claudeAPI.analyzeImageStreaming(
images: labeledImages,
systemPrompt: Self.companionVoiceResponseSystemPrompt,
systemPrompt: resolvedSystemPrompt,
conversationHistory: historyForAPI,
userPrompt: transcript,
onTextChunk: { _ in
Expand Down
72 changes: 69 additions & 3 deletions leanring-buddy/CompanionPanelView.swift
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,12 @@ struct CompanionPanelView: View {
Spacer()
.frame(height: 12)

modelPickerRow
.padding(.horizontal, 16)
VStack(spacing: 8) {
assistantModePickerRow
modelPickerRow
tftSnapshotStatusRow
}
.padding(.horizontal, 16)
}

if !companionManager.allPermissionsGranted {
Expand Down Expand Up @@ -127,7 +131,9 @@ struct CompanionPanelView: View {
@ViewBuilder
private var permissionsCopySection: some View {
if companionManager.hasCompletedOnboarding && companionManager.allPermissionsGranted {
Text("Hold Control+Option to talk.")
Text(companionManager.selectedAssistantMode == .tftCoach
? "Hold Control+Option to ask for TFT coaching."
: "Hold Control+Option to talk.")
.font(.system(size: 12, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)
.frame(maxWidth: .infinity, alignment: .leading)
Expand Down Expand Up @@ -598,6 +604,49 @@ struct CompanionPanelView: View {

// MARK: - Model Picker

private var assistantModePickerRow: some View {
HStack {
Text("Mode")
.font(.system(size: 13, weight: .medium))
.foregroundColor(DS.Colors.textSecondary)

Spacer()

HStack(spacing: 0) {
assistantModeOptionButton(mode: .general)
assistantModeOptionButton(mode: .tftCoach)
}
.background(
RoundedRectangle(cornerRadius: 6, style: .continuous)
.fill(Color.white.opacity(0.06))
)
.overlay(
RoundedRectangle(cornerRadius: 6, style: .continuous)
.stroke(DS.Colors.borderSubtle, lineWidth: 0.5)
)
}
.padding(.vertical, 4)
}

private func assistantModeOptionButton(mode: CompanionAssistantMode) -> some View {
let isSelected = companionManager.selectedAssistantMode == mode
return Button(action: {
companionManager.setSelectedAssistantMode(mode)
}) {
Text(mode.panelLabel)
.font(.system(size: 11, weight: .medium))
.foregroundColor(isSelected ? DS.Colors.textPrimary : DS.Colors.textTertiary)
.padding(.horizontal, 10)
.padding(.vertical, 5)
.background(
RoundedRectangle(cornerRadius: 5, style: .continuous)
.fill(isSelected ? Color.white.opacity(0.1) : Color.clear)
)
}
.buttonStyle(.plain)
.pointerCursor()
}

private var modelPickerRow: some View {
HStack {
Text("Model")
Expand Down Expand Up @@ -641,6 +690,23 @@ struct CompanionPanelView: View {
.pointerCursor()
}

@ViewBuilder
private var tftSnapshotStatusRow: some View {
if companionManager.selectedAssistantMode == .tftCoach {
HStack(spacing: 8) {
Image(systemName: "info.circle")
.font(.system(size: 11, weight: .medium))
.foregroundColor(DS.Colors.textTertiary)

Text(companionManager.tftMetaStatusMessage)
.font(.system(size: 10))
.foregroundColor(DS.Colors.textTertiary)
.frame(maxWidth: .infinity, alignment: .leading)
}
.padding(.vertical, 2)
}
}

// MARK: - DM Farza Button

private var dmFarzaButton: some View {
Expand Down
59 changes: 59 additions & 0 deletions leanring-buddy/TFTMetaContext.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
//
// TFTMetaContext.swift
// leanring-buddy
//
// Local, manually refreshed TFT patch/meta snapshot. This keeps the app fully
// self-contained while still giving the assistant recent TFT context.
//

import Foundation

struct TFTMetaContextSnapshot: Equatable {
let snapshotUpdatedOn: String
let latestPatchTitle: String
let latestPatchPublishedAt: String
let latestPatchURL: String
let dataDragonVersion: String
let metaNotes: [String]
}

enum TFTMetaKnowledgeBase {
// BEGIN_AUTOGEN_TFT_SNAPSHOT
static let currentSnapshot = TFTMetaContextSnapshot(
snapshotUpdatedOn: "2026-04-13",
latestPatchTitle: "Teamfight Tactics patch 16.8",
latestPatchPublishedAt: "2026-03-31T18:00:00.000Z",
latestPatchURL: "https://teamfighttactics.leagueoflegends.com/en-us/news/game-updates/teamfight-tactics-patch-16-8",
dataDragonVersion: "16.7.1",
metaNotes: [
"This is a manually maintained snapshot, not a live API feed. State confidence when meta calls are uncertain.",
"Patch notes are the source of truth for current balance direction.",
"Use the current board/shop/items shown on screen to make final recommendations.",
]
)
// END_AUTOGEN_TFT_SNAPSHOT
}

enum TFTMetaPromptBuilder {
static func buildPromptContext() -> String {
let snapshot = TFTMetaKnowledgeBase.currentSnapshot
let notes = snapshot.metaNotes.enumerated().map { index, value in
"\(index + 1). \(value)"
}.joined(separator: "\n")

return """
TFT SNAPSHOT (manual)
Snapshot last updated: \(snapshot.snapshotUpdatedOn)
Latest official patch notes: \(snapshot.latestPatchTitle) (\(snapshot.latestPatchPublishedAt))
Patch notes URL: \(snapshot.latestPatchURL)
Latest Data Dragon version: \(snapshot.dataDragonVersion)
Meta notes:
\(notes)
"""
}

static func buildStatusMessage() -> String {
let snapshot = TFTMetaKnowledgeBase.currentSnapshot
return "Manual TFT snapshot: \(snapshot.latestPatchTitle), updated \(snapshot.snapshotUpdatedOn)"
}
}
Loading