Merge/fast main 20250910 015951

.agent-os/instructions/core/aurea-codex-workflow.md

@@ -0,0 +1,24 @@
+# SYSTEM UPDATE FOR AUREA — TRUTH-FIRST CODEX WORKFLOW (NON-BYPASSABLE)
+
+**Scope:** Do not edit files directly. Output only: PLAN → DIFFS → COMMANDS → TESTS → VERIFICATION → ROLLBACK.
+
+**Contracts to enforce:**
+- NON SIMULATION CONTRACT (no persona/emotion/trust simulation). 
+- Truth-first engineering (separate DATA/REASONING/SPECULATION; surface contradictions; use ASSUMPTION when needed). 
+- Rights & Safety: refuse unsafe or rights-violating actions; propose safe alternatives. 
+
+**Output format (always):**
+1. PLAN (≤8 steps, risks/unknowns)
+2. DIFFS (fenced ```diff blocks with correct paths)
+3. COMMANDS (Codex-executable shell lines incl. git)
+4. TESTS (unit/integration proving acceptance)
+5. VERIFICATION (local/CI checklist)
+6. ROLLBACK (safe revert)
+
+**Determinism & Safety:**
+- Idempotent commands; no destructive ops without gating/dry-run.
+- Secrets via env vars only; `.env.example` updates if needed.
+- Explicit failure messages; halt on contradictions.
+
+**Acceptance criteria:** Each task must define clear, testable bullets that would fail pre-change and pass post-change.
+

.agent-os/instructions/core/codex-runbook.md

@@ -0,0 +1,31 @@
+# CODEX RUNBOOK — APPLY PATCHES WITH TRUTH KERNEL SAFETY
+
+**Objective:** Apply Aurea’s DIFFS exactly; validate with audit + tests; commit and push.
+
+## Steps (strict order)
+1) **Repo prep**
+   - `git status --porcelain`
+   - `git pull --rebase`
+   - `git checkout -b feat/<slug> || git switch feat/<slug>`
+2) **Apply patches**
+   - For each ```diff block from Aurea, apply using:
+     - `git apply --index --reject`  # fail on rejects
+   - Ensure any new directories exist before apply.
+3) **Install & build**
+   - `npm ci`
+   - `npm run build || true`  # only if a build script exists
+4) **Static checks & tests**
+   - `npm test`
+   - `node backend/scripts/audit-truth.cjs`
+5) **Dry-run and (if required) safe execute**
+   - If `sf` exists: `sf dry`  # never run `sf apply --apply` unless explicitly requested
+6) **Commit & push**
+   - `git add -A`
+   - `git commit -m "feat: <concise summary of change>"`
+   - `git push -u origin HEAD`
+7) **PR**
+   - Create PR titled `feat: <summary> (TruthKernel)`
+   - PR body includes acceptance-criteria checklist and last audit/test outputs.
+
+**Failure policy:** On ANY failure (apply/test/audit), stop and post the failing command, exit code, and last 120 lines of logs. No partial commits.
+

.agent-os/specs/1757008963219-jina-ai-for-retrieval-rerank-over-workspace-docs-https-do/spec.md

@@ -0,0 +1,64 @@
+# spec.md
+
+## Intent
+Build a UK-focused semantic search API that integrates Jina AI's embedding and reranking services with TruthLens verification, enabling RAG-powered document search with fact-checking capabilities for workspace knowledge bases.
+
+## Acceptance Criteria
+1. Successfully index 100+ workspace documents via Jina Reader API
+2. Search response time <300ms for 95% of queries
+3. Rerank top-10 results with >85% relevance accuracy
+4. TruthLens confidence scores correlate with rerank scores (r>0.7)
+5. API handles 100 concurrent requests without degradation
+6. All responses include source citations and confidence metrics
+
+## Thin-Slice MVP
+A REST API endpoint that accepts a search query, embeds it using Jina, searches a vector store, reranks results, and returns TruthLens-verified responses with citations.
+
+```
+POST /api/v1/search
+{
+  "query": "How does TruthLens verify facts?",
+  "limit": 5,
+  "verify": true
+}
+→ Returns ranked, verified results with confidence scores
+```
+
+## Tasks (≤12)
+
+### Setup [2h]
+1. Configure Jina API credentials and rate limits in `.env`
+2. Initialize PostgreSQL with pgvector extension for embeddings
+
+### Core Implementation [8h]
+3. Create `JinaAdapter` class with embed/rerank methods
+4. Build document chunker (1000 tokens, 200 overlap)
+5. Implement batch document indexer with progress tracking
+6. Create vector search function with similarity threshold
+7. Build reranking pipeline for top-20 candidates
+8. Implement TruthLens verification against indexed sources
+
+### API & Integration [4h]
+9. Create FastAPI endpoint with request/response models
+10. Add caching layer for frequent embeddings (Redis TTL=1h)
+11. Implement async processing for parallel operations
+
+### Testing & Deploy [2h]
+12. Write integration tests with mock documents and deploy to Railway/Render
+
+## Week 1 Metrics
+- **Indexing Volume**: 500+ documents processed
+- **Query Latency**: P95 <300ms, P50 <150ms
+- **Relevance Score**: >85% for test query set
+- **API Uptime**: >99% availability
+- **Cost Efficiency**: <£50 total API spend
+- **User Feedback**: 10+ beta testers provide relevance ratings
+
+## Run (approved)
+echo "[spec] sanity checks"
+ls -alh /home/michael/soulfield/workspace/specs
+echo "[policy] show TruthLens rules"
+head -n 20 /home/michael/soulfield/workspace/knowledge/TruthLens.md
+echo "[index] first 10 lines"
+head -n 10 /home/michael/soulfield/workspace/data/index.json
+

.agent-os/standards/CODEX RUNBOOK APPLY PATCHES.txt

@@ -0,0 +1,37 @@
+CODEX RUNBOOK — APPLY PATCHES WITH TRUTH KERNEL SAFETY
+
+Objective: Apply Aurea’s DIFFS exactly; validate with audit + tests; commit and push.
+
+Steps (strict order):
+1) Repo prep
+   - git status --porcelain
+   - git pull --rebase
+   - git checkout -b feat/<slug> || git switch feat/<slug>
+
+2) Apply patches
+   - For each ```diff block from Aurea, apply using `git apply --index --reject` (fail on rejects).
+   - If new files: ensure directories exist before apply.
+
+3) Install & build
+   - npm ci
+   - npm run build || true  # only if build script exists
+
+4) Static checks & tests
+   - npm test
+   - node backend/scripts/audit-truth.cjs
+
+5) Dry-run and (if required) safe execute
+   - if `sf` exists: `sf dry`  # do not run `sf apply --apply` unless explicitly requested
+
+6) Commit & push
+   - git add -A
+   - git commit -m "feat: <concise summary of change>"
+   - git push -u origin HEAD
+
+7) PR (if platform supports it)
+   - Create PR titled "feat: <summary> (TruthKernel)"
+   - PR body: include checklist of acceptance criteria and audit/test outputs.
+
+Failure policy:
+- On ANY failure (apply/test/audit), stop and post the failing command, exit code, and last 120 lines of logs. No partial commits.
+

.github/pull_request_template.md

@@ -1,20 +1,12 @@
-Title: <short summary>
+## Summary
+Describe the change in one or two sentences.
 
-Summary
-- What changed and why (1–3 lines).
+## Checklist (Truth Kernel)
+- [ ] Local: `npm ci`
+- [ ] Local: `node backend/scripts/audit-truth.cjs` (no violations)
+- [ ] Local: `npm test` (Node’s built-in runner)
+- [ ] Docs updated if behavior changes (README / STATUS / Knowledge)
 
-Checklist
-- [ ] Scope is minimal and focused
-- [ ] CI green locally (if applicable)
-- [ ] Tests added/updated or not applicable
-- [ ] Docs updated or not applicable
-
-Validation
-- Steps to verify manually:
-  1. 
-  2. 
-
-Links
-- Related issue: #
-- Context/spec: 
+## Notes
+Link to related specs, runs, or artifacts if relevant.
 

.github/workflows/ci.yml

@@ -2,40 +2,27 @@ name: CI
 
 on:
   push:
-    branches: [ main ]
+    branches: [ main, feat/**, fix/**, docs/**, chore/** ]
   pull_request:
     branches: [ main ]
 
 jobs:
   node:
-    if: ${{ hashFiles('package.json') != '' }}
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
         uses: actions/checkout@v4
-      - name: Setup Node
+      - name: Setup Node.js
         uses: actions/setup-node@v4
         with:
           node-version: '20'
           cache: 'npm'
-      - name: Install
+      - name: Install deps
         run: npm ci
-      - name: API health (offline)
-        run: |
-          DEV_NO_API=1 node backend/index.cjs &
-          echo $! > api.pid
-          sleep 1
-          curl -fsS http://127.0.0.1:8790/health | tee health.json
-          kill $(cat api.pid)
-      - name: Run ESLint (npx)
-        run: npx -y eslint .
-      - name: Lint (if present)
-        run: npm run -s lint --if-present
-      - name: Test (if present)
-        run: npm run -s test --if-present
+      - name: Run tests (node:test)
+        run: npm test
 
   python:
-    if: ${{ hashFiles('requirements.txt') != '' || hashFiles('pyproject.toml') != '' }}
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
@@ -44,17 +31,5 @@ jobs:
         uses: actions/setup-python@v5
         with:
           python-version: '3.11'
-      - name: Install deps (if requirements.txt)
-        run: |
-          if [ -f requirements.txt ]; then
-            python -m pip install --upgrade pip
-            pip install -r requirements.txt
-          fi
-      - name: Run pytest if tests are present
-        run: |
-          if ls -1 tests test_*.py 2>/dev/null | grep -q .; then
-            pip install pytest
-            pytest -q
-          else
-            echo "No python tests found; skipping"
-          fi
+      - name: Sanity
+        run: python --version

.github/workflows/truth-kernel.yml

@@ -0,0 +1,29 @@
+name: truth-kernel
+
+on:
+  push:
+    branches: [ main, feat/** ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          cache: 'npm'
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Truth Kernel audit
+        run: node backend/scripts/audit-truth.cjs
+
+      - name: Unit tests
+        run: npm test