Merge/feat context spine embeddings

.agent-os/instructions/core/aurea-codex-workflow.md

@@ -0,0 +1,24 @@
+# SYSTEM UPDATE FOR AUREA — TRUTH-FIRST CODEX WORKFLOW (NON-BYPASSABLE)
+
+**Scope:** Do not edit files directly. Output only: PLAN → DIFFS → COMMANDS → TESTS → VERIFICATION → ROLLBACK.
+
+**Contracts to enforce:**
+- NON SIMULATION CONTRACT (no persona/emotion/trust simulation). 
+- Truth-first engineering (separate DATA/REASONING/SPECULATION; surface contradictions; use ASSUMPTION when needed). 
+- Rights & Safety: refuse unsafe or rights-violating actions; propose safe alternatives. 
+
+**Output format (always):**
+1. PLAN (≤8 steps, risks/unknowns)
+2. DIFFS (fenced ```diff blocks with correct paths)
+3. COMMANDS (Codex-executable shell lines incl. git)
+4. TESTS (unit/integration proving acceptance)
+5. VERIFICATION (local/CI checklist)
+6. ROLLBACK (safe revert)
+
+**Determinism & Safety:**
+- Idempotent commands; no destructive ops without gating/dry-run.
+- Secrets via env vars only; `.env.example` updates if needed.
+- Explicit failure messages; halt on contradictions.
+
+**Acceptance criteria:** Each task must define clear, testable bullets that would fail pre-change and pass post-change.
+

.agent-os/instructions/core/codex-runbook.md

@@ -0,0 +1,31 @@
+# CODEX RUNBOOK — APPLY PATCHES WITH TRUTH KERNEL SAFETY
+
+**Objective:** Apply Aurea’s DIFFS exactly; validate with audit + tests; commit and push.
+
+## Steps (strict order)
+1) **Repo prep**
+   - `git status --porcelain`
+   - `git pull --rebase`
+   - `git checkout -b feat/<slug> || git switch feat/<slug>`
+2) **Apply patches**
+   - For each ```diff block from Aurea, apply using:
+     - `git apply --index --reject`  # fail on rejects
+   - Ensure any new directories exist before apply.
+3) **Install & build**
+   - `npm ci`
+   - `npm run build || true`  # only if a build script exists
+4) **Static checks & tests**
+   - `npm test`
+   - `node backend/scripts/audit-truth.cjs`
+5) **Dry-run and (if required) safe execute**
+   - If `sf` exists: `sf dry`  # never run `sf apply --apply` unless explicitly requested
+6) **Commit & push**
+   - `git add -A`
+   - `git commit -m "feat: <concise summary of change>"`
+   - `git push -u origin HEAD`
+7) **PR**
+   - Create PR titled `feat: <summary> (TruthKernel)`
+   - PR body includes acceptance-criteria checklist and last audit/test outputs.
+
+**Failure policy:** On ANY failure (apply/test/audit), stop and post the failing command, exit code, and last 120 lines of logs. No partial commits.
+

.agent-os/specs/1757008963219-jina-ai-for-retrieval-rerank-over-workspace-docs-https-do/spec.md

@@ -0,0 +1,64 @@
+# spec.md
+
+## Intent
+Build a UK-focused semantic search API that integrates Jina AI's embedding and reranking services with TruthLens verification, enabling RAG-powered document search with fact-checking capabilities for workspace knowledge bases.
+
+## Acceptance Criteria
+1. Successfully index 100+ workspace documents via Jina Reader API
+2. Search response time <300ms for 95% of queries
+3. Rerank top-10 results with >85% relevance accuracy
+4. TruthLens confidence scores correlate with rerank scores (r>0.7)
+5. API handles 100 concurrent requests without degradation
+6. All responses include source citations and confidence metrics
+
+## Thin-Slice MVP
+A REST API endpoint that accepts a search query, embeds it using Jina, searches a vector store, reranks results, and returns TruthLens-verified responses with citations.
+
+```
+POST /api/v1/search
+{
+  "query": "How does TruthLens verify facts?",
+  "limit": 5,
+  "verify": true
+}
+→ Returns ranked, verified results with confidence scores
+```
+
+## Tasks (≤12)
+
+### Setup [2h]
+1. Configure Jina API credentials and rate limits in `.env`
+2. Initialize PostgreSQL with pgvector extension for embeddings
+
+### Core Implementation [8h]
+3. Create `JinaAdapter` class with embed/rerank methods
+4. Build document chunker (1000 tokens, 200 overlap)
+5. Implement batch document indexer with progress tracking
+6. Create vector search function with similarity threshold
+7. Build reranking pipeline for top-20 candidates
+8. Implement TruthLens verification against indexed sources
+
+### API & Integration [4h]
+9. Create FastAPI endpoint with request/response models
+10. Add caching layer for frequent embeddings (Redis TTL=1h)
+11. Implement async processing for parallel operations
+
+### Testing & Deploy [2h]
+12. Write integration tests with mock documents and deploy to Railway/Render
+
+## Week 1 Metrics
+- **Indexing Volume**: 500+ documents processed
+- **Query Latency**: P95 <300ms, P50 <150ms
+- **Relevance Score**: >85% for test query set
+- **API Uptime**: >99% availability
+- **Cost Efficiency**: <£50 total API spend
+- **User Feedback**: 10+ beta testers provide relevance ratings
+
+## Run (approved)
+echo "[spec] sanity checks"
+ls -alh /home/michael/soulfield/workspace/specs
+echo "[policy] show TruthLens rules"
+head -n 20 /home/michael/soulfield/workspace/knowledge/TruthLens.md
+echo "[index] first 10 lines"
+head -n 10 /home/michael/soulfield/workspace/data/index.json
+

.agent-os/standards/CODEX RUNBOOK APPLY PATCHES.txt

@@ -0,0 +1,37 @@
+CODEX RUNBOOK — APPLY PATCHES WITH TRUTH KERNEL SAFETY
+
+Objective: Apply Aurea’s DIFFS exactly; validate with audit + tests; commit and push.
+
+Steps (strict order):
+1) Repo prep
+   - git status --porcelain
+   - git pull --rebase
+   - git checkout -b feat/<slug> || git switch feat/<slug>
+
+2) Apply patches
+   - For each ```diff block from Aurea, apply using `git apply --index --reject` (fail on rejects).
+   - If new files: ensure directories exist before apply.
+
+3) Install & build
+   - npm ci
+   - npm run build || true  # only if build script exists
+
+4) Static checks & tests
+   - npm test
+   - node backend/scripts/audit-truth.cjs
+
+5) Dry-run and (if required) safe execute
+   - if `sf` exists: `sf dry`  # do not run `sf apply --apply` unless explicitly requested
+
+6) Commit & push
+   - git add -A
+   - git commit -m "feat: <concise summary of change>"
+   - git push -u origin HEAD
+
+7) PR (if platform supports it)
+   - Create PR titled "feat: <summary> (TruthKernel)"
+   - PR body: include checklist of acceptance criteria and audit/test outputs.
+
+Failure policy:
+- On ANY failure (apply/test/audit), stop and post the failing command, exit code, and last 120 lines of logs. No partial commits.
+

.chatgpt/commands/brain-dump-analysis.md

@@ -0,0 +1,23 @@
+# /brain-dump-analysis
+
+Purpose: synthesize braindumps into themes, questions, breached rules, and realizations.
+
+Process:
+- Read `braindumps/` markdown files.
+- Extract keywords and frequent phrases.
+- Identify questions (“?”), and lines like “rule(s) I keep breaking…”.
+- Output ASCII mind map + bullet lists.
+
+Output: `braindumps/analysis/insights-YYYY-MM-DD.md`
+
+## Usage
+
+Ensure you have markdown files under `braindumps/`, then run:
+
+```
+node tools/brain-dump-analysis.cjs
+# or
+node tools/sf.cjs brain-dump-analysis
+```
+
+Generates: `braindumps/analysis/insights-<today>.md` containing themes, questions, and a simple ASCII mind map.

.chatgpt/commands/daily-brief.md

@@ -0,0 +1,24 @@
+# /daily-brief (markets)
+
+Purpose: structured market preparation without external calls.
+
+Sources:
+- Local workspace markdowns (keywords/interests)
+
+Content:
+- Sessions for ES + majors FX (placeholders OK)
+- Checklist: econ calendar, overnight ranges, ATR%, key levels
+- Why-each-item-matters notes
+- Action checklist slots
+
+Output: `briefs/daily-brief-YYYY-MM-DD.md`
+
+## Usage
+
+```
+node tools/daily-brief.cjs
+# or
+node tools/sf.cjs daily-brief
+```
+
+The brief is structure-only (no web calls). Fill checklist items manually.

.chatgpt/commands/daily-checkin.md

@@ -0,0 +1,34 @@
+# /daily-checkin
+
+Purpose: end-of-day reflection and planning.
+
+Prompts:
+- mood 1–10, energy 1–10
+- top 3 wins
+- tomorrow’s #1
+- blockers
+
+Outputs:
+- `journal/daily/YYYY-MM-DD.md`
+- `journal/daily/YYYY-MM-DD-reflection.md` (trend of last 3 days, momentum score, suggestions, best time block)
+
+## Usage
+
+Run from repo root (Node ≥ 20):
+
+```
+node tools/daily-checkin.cjs
+# or
+node tools/sf.cjs daily-checkin
+```
+
+Non-interactive (smoke) via env (good for tests/automation):
+
+```
+TEST_INPUT='{"mood":7,"energy":6,"wins":"win1;win2;win3","tomorrow1":"Top focus","blockers":"X"}' \
+  node tools/daily-checkin.cjs
+```
+
+Files written:
+- `journal/daily/<today>.md`
+- `journal/daily/<today>-reflection.md`

.chatgpt/commands/weekly-checkin.md

@@ -0,0 +1,39 @@
+# /weekly-checkin
+
+Purpose: capture weekly TRADING + STUDY metrics and generate a compact report.
+
+Prompts to collect:
+- TRADING: R values list, regime tag, volatility proxy.
+- STUDY: focused hours, problems solved, PRs merged, tests added, coverage %, commits, learning sprints, notes.
+
+Calculations:
+- Expectancy (R) = average(R)
+- Win rate = wins / N
+- Max drawdown (R) = min cumulative R trough
+- Kelly-lite (informational) = winrate – (1 – winrate)/(avg_win_R/|avg_loss_R|)
+- Study velocity = focused_hours + problems_solved*0.5 + PRs*0.5
+
+Outputs:
+- Append JSON block to `metrics/metrics-history.md`
+- Create `metrics/weekly-report-YYYY-MM-DD.md` with tables, deltas, emoji signals, and 3–5 recommendations.
+
+## Usage
+
+Interactive:
+
+```
+node tools/weekly-checkin.cjs
+# or
+node tools/sf.cjs weekly-checkin
+```
+
+Non-interactive (CI/smoke):
+
+```
+TEST_INPUT='{"R_list":"1 -0.5 0.7","focused_hours":5,"problems_solved":4,"prs_merged":2}' \
+  node tools/weekly-checkin.cjs
+```
+
+Artifacts:
+- `metrics/metrics-history.md` (appended block)
+- `metrics/weekly-report-<today>.md`

.chatgpt/subagents/daily-reflection.md

@@ -0,0 +1,9 @@
+System: You are ChatGPT daily reflection coach. Keep tone supportive and brief.
+
+Input: mood (1–10), energy (1–10), top wins, blockers, last 3 days trends.
+Output:
+- Tiny trend for mood/energy (up/down/flat)
+- Accomplishment momentum score
+- 2–3 gentle suggestions
+- Best time block for tomorrow (AM if trend up, PM otherwise)
+

.chatgpt/subagents/metrics-analyst.md

@@ -0,0 +1,8 @@
+System: You are ChatGPT metrics analyst. Be concise, numerical, and actionable. Use tables, deltas, and emoji signals (🚀 📈 📉 ➡️).
+
+Given weekly TRADING R data and STUDY metrics, compute:
+- Expectancy (R), Win rate, Max drawdown (R)
+- Kelly-lite signal (informational only)
+- Study velocity
+Then produce 3–5 concrete recommendations.
+

.env.example

@@ -1,15 +1,70 @@
-## Copy to `.env` and fill values
+## Copy this file to `.env` and fill in values.
+## Only `.env` is loaded at runtime; this is just a template.
 
-# Core
+# ---- Core (Aiden / API server) ----
+# Anthropic key (preferred env var name)
+ANTHROPIC_API_KEY=
+# Legacy/alternate name (also supported)
 CLAUDE_API_KEY=
-AIDEN_MODEL=
+# Claude model used by Aiden
+AIDEN_MODEL=claude-opus-4.1-20250805
+# Run offline when 1 (returns "dev offline"); set 0 for live API calls
+DEV_NO_API=0
+# HTTP port for backend/index.cjs
+PORT=8790
+CONTEXT_SPINE=0
+
+# ---- Embeddings ----
+# Provider for text embeddings used by memory and context spine
+EMBED_PROVIDER=local                 # local | openai
+# Model name for OpenAI provider only
+EMBED_MODEL=text-embedding-3-small   # used only if EMBED_PROVIDER=openai
+# Heartbeat embedding throttle (embed 1 of every N heartbeats)
+HEARTBEAT_EMBED_EVERY=4              # embed one of every N heartbeats
+OBSIDIAN_DIR=./workspace/docs/Obsidian
+NOTES_DIR=./workspace/docs/Obsidian
+
+# Retention (days) for context items (heartbeats/calendar past events)
+CONTEXT_RETENTION_DAYS=14
+
+# ---- Vector Memory (Pinecone + embeddings) ----
+# Enable Pinecone-backed memory (1 to enable)
+USE_PINECONE=0
+# Pinecone credentials and index settings
+PINECONE_API_KEY=
+PINECONE_INDEX=soulfield-memory
+PINECONE_CLOUD=aws
+PINECONE_REGION=us-east-1
+# OpenAI API key for embeddings (1536-dim)
 OPENAI_API_KEY=
+# Optional: batch size for upserts
+MEMORY_UPSERT_BATCH=64
 
-# Optional adapters
+# ---- Adapters ----
+# Jina API (embeddings/rerank)
 JINA_API_KEY=
+# Default embedding model for tools/test-memory-suite.cjs
+JINA_MODEL=jina-embeddings-v2-base-en
+
+# InfraNodus API
 INFRANODUS_API_KEY=
+# Optional custom base URL
+INFRANODUS_API_BASE=https://api.infranodus.com
 
-# Vector memory (optional)
-USE_PINECONE=0
-PINECONE_API_KEY=
+# Bright Data (scraper)
+BRIGHTDATA_TOKEN=
+
+# ---- Misc / Tooling ----
+# GitHub automation scripts
+GITHUB_TOKEN=
+# Optional hint if you swap embedding providers (not read by current code)
+# EMBED_PROVIDER=openai
 
+# ---- Google Calendar (OAuth2) ----
+# Used by contextSpine.js for calendar sync
+GCAL_CLIENT_ID=
+GCAL_CLIENT_SECRET=
+GCAL_REDIRECT_URI=
+GCAL_REFRESH_TOKEN=
+GCAL_WINDOW_HOURS=24
+GCAL_CALENDAR_ID=primary