Feat/readme refresh

README.md

@@ -1,136 +1,65 @@
 # Soulfield OS
 
-A backend‑first agent OS for research → gap analysis → spec → (safe) execution. It exposes a minimal HTTP API, a read‑only MCP filesystem tool, a terminal spec browser, and adapters for Claude (Aiden), Jina rerank, InfraNodus gap analysis, and a policy‑first scraper.
-
-## Features
-- API server: `/health`, `/chat` with “!” command router and `@agent` council routing.
-- Agents: Aiden (Claude), Jina (semantic rerank), InfraNodus (gap analysis), Scraper (allowlist only).
-- Memory: File memory by default; optional Pinecone with OpenAI embeddings.
-- Compliance: TruthLens shim wraps outputs; scraper domain allowlist; read‑only MCP filesystem server.
-- TUI: Terminal UI to browse/edit specs in `.agent-os/specs/*/spec.md`.
-- Safe execution: `!coder-apply` executes only whitelisted commands from approved spec sections.
-- Research enhancements: Bright Data templates (e.g., `google/serp`, `ebay/search`) and a discovery adapter (multi‑engine + BD SERP) to broaden coverage while staying policy‑first.
-
-## Quickstart
-- Requirements: Node.js 18+ (20+ recommended). Create `.env` (see below). Do not commit real keys.
-- Install: `npm install`
-- Start API: `npm start`
-  - Health: `curl -s http://127.0.0.1:8790/health`
-  - Help: `curl -s -X POST http://127.0.0.1:8790/chat -H 'content-type: application/json' -d '{"prompt":"!help"}'`
-  - Aiden: `curl -s -X POST http://127.0.0.1:8790/chat -H 'content-type: application/json' -d '{"prompt":"@aiden: Summarize the workspace goals"}'`
-- MCP (read‑only FS): `npm run start:mcp`
-  - Tools: GET `http://127.0.0.1:8791/mcp/tools`
-  - Call: POST `http://127.0.0.1:8791/mcp/call` `{ "name":"list_dir", "args": {"path":"workspace"} }`
-- Spec TUI: `npm run start:tui`
-- Spec apply tests: `npm run test:apply` (dry) or `npm run test:apply:run` (execute whitelisted commands)
-
-## Environment
-Set in `.env`:
-- Claude: `CLAUDE_API_KEY`, `AIDEN_MODEL`
-- Jina: `JINA_API_KEY`, `JINA_MODEL`
-- Pinecone (optional): `USE_PINECONE=1`, `PINECONE_API_KEY`, `PINECONE_INDEX`
-- OpenAI Embeddings (if Pinecone): `OPENAI_API_KEY`
-- InfraNodus: `INFRANODUS_API_KEY`, `INFRANODUS_API_BASE`
-- Bright Data (optional): `BRIGHTDATA_TOKEN`, `BD_BASE_URL=https://api.brightdata.com`, `BD_TIMEOUT_MS=30000`
-
-## Commands (via /chat)
-- `!help` – menu
-- `!note <text> #tags` / `!recall #tag` – capture/recall
-- `!plan-add active|future "Name" #tags` / `!plan-list` – roadmap
-- `!golden "desc" #tags` / `!golden-list [#tag] [N]` – golden ideas
-- `!session-note "text" #tags` / `!session-list [#tag] [N]` – session timeline
-- `!learn <text> #tags` / `!knowledge-list [#tag] [N]` – knowledge
-- `!learn-file <relative/path> [#tags]` – ingest file
-- `!coder-apply [--spec <id>] [#apply]` – safe execute from latest/given spec
-
-CLI (local)
-- `tools/sf specify` – generate spec from newest research
-- `tools/sf specify:infra [exports/<file>.json]` – generate spec from InfraNodus export
-- `tools/sf dry` / `tools/sf apply --apply` – safe preview/execute Run blocks
-- `tools/sf search "<query>"` (planned) – discovery via multiple engines + Bright Data SERP → `workspace/data/search/*.jsonl`
-
-## Paths and Data
-- API: `backend/index.cjs`
-- Jobs: `backend/jobs.js`
-- Council: `backend/council.js`
-- Memory (file): `memory.js` → `data/memory.json`
-- Memory (Pinecone): `backend/services/memory/memory-pinecone.cjs`
-- Scraper: `backend/services/scraper/index.cjs` → `workspace/data/scrapes/*.jsonl`
-- Specs: `backend/.agent-os/specs/<id>/spec.md`
-- Apply logs: `backend/.agent-os/runs/*.log`
-- MCP: `backend/mcp-server.cjs`
-- TUI: `backend/tui.js`
-
-## Security & Compliance
-- TruthLens wraps outputs (see `backend/truthLens.js` / `truthlens.cjs`).
-- Scraper requires allowlist (`backend/services/scraper/config/allowlist.yaml`) using template tokens (e.g., `bd:google/serp`), tiered domains (gov, marketplaces, media, vendor_docs), bounded wildcards, and purpose tags.
-- MCP is read-only and jailed to project root.
-- `!coder-apply` whitelists basic commands only (`echo`, `ls`, `cat`, `head`).
-
-## CI / Truth Kernel (non-bypassable)
-- Workflow: `.github/workflows/truth-kernel.yml` runs on pushes and PRs.
-- Environment: Ubuntu, Node.js 20 (matches local stack).
-- Steps:
-  1. `npm ci`
-  2. `node backend/scripts/audit-truth.cjs`
-  3. `npm test`  ← runs Node’s built-in test runner (`node:test`) against `backend/tests/*.test.cjs`
-- If any step fails, the merge is blocked. This ensures only TruthLens-compliant, non-simulation, and tested changes land on `main`.
-
-## Using Codex (file-based prompt)
-- Canonical runbook lives at `.agent-os/instructions/core/codex-runbook.md`.
-- Invoke Codex with a file-based system prompt flag (example):
-  ```bash
-  codex run --prompt-file .agent-os/instructions/core/codex-runbook.md
-  ```
-- Aurea outputs PLAN → DIFFS → COMMANDS → TESTS → VERIFICATION → ROLLBACK; Codex applies the diffs and pushes per the runbook.
-
-## Known Issues / Notes
-- Rotate any real API keys in `.env` before sharing.
-- InfraNodus client stubs queue requests offline; integrate the real API when keys are set.
-- Pinecone path is optional; with `USE_PINECONE!=1`, the system uses file memory.
-- Bright Data integration is template‑based; do not enable all categories by default.
-
-## Soulfield OS — Docs Zone
-
-## TruthLens (runtime + repo gate)
-All assistant and agent outputs are wrapped by TruthLens at runtime, enforcing structural clarity and non-simulation. The canonical shim lives at `truthLens.js`; the backend module is `backend/truthlens.cjs`.
-
-Repo hard-gate: CI (`.github/workflows/truth-kernel.yml`) blocks merges unless the Truth audit and tests pass. The audit script validates that generated artifacts either:
-- contain `meta.lens.passed` with `"structure"` when JSON, or
-- contain no simulation phrases in prose (code blocks are ignored).
-
-Run locally:
+> The AI Business OS That Doesn’t Hallucinate  
+> Orchestrates research → strategy → execution → analytics, all gated by TruthLens.
+
+---
+
+## ✨ What It Is
+Soulfield OS is a backend-first orchestration system where:
+- Aiden (Claude) turns research into specs.
+- InfraNodus identifies content/market gaps.
+- Bright Data Scraper fetches compliant marketplace/search data (allowlist enforced).
+- Jina reranks and semantically searches across workspace docs.
+- TruthLens checkpoints every step, blocking simulation, hallucinations, or invalid metrics.
+
+Outputs: trusted, auditable business blueprints.
+
+---
+
+## 🚀 Quickstart
+
 ```bash
+git clone https://github.com/mrhpython/Soulfield
+cd Soulfield
 npm ci
-node backend/scripts/audit-truth.cjs
-npm test
+
+# Research loop
+sf research "eco-friendly digital planners UK"
+sf specify
+sf index
+sf dry
+sf apply --apply
 ```
 
-Policy source of truth: `workspace/knowledge/TruthLens.md`.
+---
 
-## Codex-first workflow
-We prefer plans → diffs → commands → tests → verification → rollback. Codex applies patches and pushes. See runbook:
-```
-.agent-os/instructions/core/codex-runbook.md
-```
+## 🛡️ Compliance
+- Default jurisdiction: UK, currency GBP.
+- Online-first businesses only.
+- Scraping only from allowlisted templates/domains (`backend/services/scraper/config/allowlist.yaml`).
+- No login-wall, PII, or policy-violating sources.
+- TruthLens audit enforced in CI: `node backend/scripts/audit-truth.cjs`.
+
+---
 
-## Quick links
-- Runtime lens: `truthLens.js`, `backend/truthlens.cjs`
-- Audit script: `backend/scripts/audit-truth.cjs`
-- Tests: `backend/tests/*.test.cjs` (Node’s built-in `node:test`)
-- Ops status & commands: `workspace/docs/STATUS.md`
+## 📚 Docs
+- [Design Intent](workspace/docs/Soulfield%20OS%20Design%20Intent.md)
+- [STATUS.md](workspace/docs/STATUS.md)
+- [TruthLens Policy](workspace/knowledge/TruthLens.md)
+- [TruthLens Vision](workspace/knowledge/TruthLens-Vision.md)
 
-## License
-See repository terms or your organization policy. No license header added by default.
-### Local testing (Ubuntu)
-- Run all unit tests: `npm test`
-- Run the Truth audit only: `node backend/scripts/audit-truth.cjs`
+---
 
-## PR Titles/Bodies — Docs Zone
+## 🛠️ Development
+- Node.js ≥20
+- CLI helper: `sf` (research/specify/apply/log)
+- CI: TruthKernel audit + Node/Python checks
+- Codex workflow: PLAN → DIFFS → COMMANDS → TESTS → VERIFICATION → ROLLBACK
 
-- **Merge Runbook + Helper Reapply**
-  - Title: `docs: reapply merge runbook + helper scripts`
-  - Body: `Re-adds PR helper (tools/gh-pr-open-and-merge.cjs) and batch script (tools/merge-three-prs.sh), and marks the batch executable. No runtime changes.`
-  - Labels: `docs,tools`
+---
 
-> Always reuse these strings when reopening or reapplying the merge runbook/PR helper. Do not invent new wording — keeps history clean and consistent.
+## 📌 One-Liner
+Soulfield OS = orchestration hub.  
+TruthLens = trust filter.  
+Together → reliable AI business operations.