diff --git a/README.md b/README.md index 1d33db53..77df2b12 100644 --- a/README.md +++ b/README.md @@ -1,136 +1,65 @@ # Soulfield OS -A backend‑first agent OS for research → gap analysis → spec → (safe) execution. It exposes a minimal HTTP API, a read‑only MCP filesystem tool, a terminal spec browser, and adapters for Claude (Aiden), Jina rerank, InfraNodus gap analysis, and a policy‑first scraper. - -## Features -- API server: `/health`, `/chat` with “!” command router and `@agent` council routing. -- Agents: Aiden (Claude), Jina (semantic rerank), InfraNodus (gap analysis), Scraper (allowlist only). -- Memory: File memory by default; optional Pinecone with OpenAI embeddings. -- Compliance: TruthLens shim wraps outputs; scraper domain allowlist; read‑only MCP filesystem server. -- TUI: Terminal UI to browse/edit specs in `.agent-os/specs/*/spec.md`. -- Safe execution: `!coder-apply` executes only whitelisted commands from approved spec sections. -- Research enhancements: Bright Data templates (e.g., `google/serp`, `ebay/search`) and a discovery adapter (multi‑engine + BD SERP) to broaden coverage while staying policy‑first. - -## Quickstart -- Requirements: Node.js 18+ (20+ recommended). Create `.env` (see below). Do not commit real keys. -- Install: `npm install` -- Start API: `npm start` - - Health: `curl -s http://127.0.0.1:8790/health` - - Help: `curl -s -X POST http://127.0.0.1:8790/chat -H 'content-type: application/json' -d '{"prompt":"!help"}'` - - Aiden: `curl -s -X POST http://127.0.0.1:8790/chat -H 'content-type: application/json' -d '{"prompt":"@aiden: Summarize the workspace goals"}'` -- MCP (read‑only FS): `npm run start:mcp` - - Tools: GET `http://127.0.0.1:8791/mcp/tools` - - Call: POST `http://127.0.0.1:8791/mcp/call` `{ "name":"list_dir", "args": {"path":"workspace"} }` -- Spec TUI: `npm run start:tui` -- Spec apply tests: `npm run test:apply` (dry) or `npm run test:apply:run` (execute whitelisted commands) - -## Environment -Set in `.env`: -- Claude: `CLAUDE_API_KEY`, `AIDEN_MODEL` -- Jina: `JINA_API_KEY`, `JINA_MODEL` -- Pinecone (optional): `USE_PINECONE=1`, `PINECONE_API_KEY`, `PINECONE_INDEX` -- OpenAI Embeddings (if Pinecone): `OPENAI_API_KEY` -- InfraNodus: `INFRANODUS_API_KEY`, `INFRANODUS_API_BASE` -- Bright Data (optional): `BRIGHTDATA_TOKEN`, `BD_BASE_URL=https://api.brightdata.com`, `BD_TIMEOUT_MS=30000` - -## Commands (via /chat) -- `!help` – menu -- `!note #tags` / `!recall #tag` – capture/recall -- `!plan-add active|future "Name" #tags` / `!plan-list` – roadmap -- `!golden "desc" #tags` / `!golden-list [#tag] [N]` – golden ideas -- `!session-note "text" #tags` / `!session-list [#tag] [N]` – session timeline -- `!learn #tags` / `!knowledge-list [#tag] [N]` – knowledge -- `!learn-file [#tags]` – ingest file -- `!coder-apply [--spec ] [#apply]` – safe execute from latest/given spec - -CLI (local) -- `tools/sf specify` – generate spec from newest research -- `tools/sf specify:infra [exports/.json]` – generate spec from InfraNodus export -- `tools/sf dry` / `tools/sf apply --apply` – safe preview/execute Run blocks -- `tools/sf search ""` (planned) – discovery via multiple engines + Bright Data SERP → `workspace/data/search/*.jsonl` - -## Paths and Data -- API: `backend/index.cjs` -- Jobs: `backend/jobs.js` -- Council: `backend/council.js` -- Memory (file): `memory.js` → `data/memory.json` -- Memory (Pinecone): `backend/services/memory/memory-pinecone.cjs` -- Scraper: `backend/services/scraper/index.cjs` → `workspace/data/scrapes/*.jsonl` -- Specs: `backend/.agent-os/specs//spec.md` -- Apply logs: `backend/.agent-os/runs/*.log` -- MCP: `backend/mcp-server.cjs` -- TUI: `backend/tui.js` - -## Security & Compliance -- TruthLens wraps outputs (see `backend/truthLens.js` / `truthlens.cjs`). -- Scraper requires allowlist (`backend/services/scraper/config/allowlist.yaml`) using template tokens (e.g., `bd:google/serp`), tiered domains (gov, marketplaces, media, vendor_docs), bounded wildcards, and purpose tags. -- MCP is read-only and jailed to project root. -- `!coder-apply` whitelists basic commands only (`echo`, `ls`, `cat`, `head`). - -## CI / Truth Kernel (non-bypassable) -- Workflow: `.github/workflows/truth-kernel.yml` runs on pushes and PRs. -- Environment: Ubuntu, Node.js 20 (matches local stack). -- Steps: - 1. `npm ci` - 2. `node backend/scripts/audit-truth.cjs` - 3. `npm test` ← runs Node’s built-in test runner (`node:test`) against `backend/tests/*.test.cjs` -- If any step fails, the merge is blocked. This ensures only TruthLens-compliant, non-simulation, and tested changes land on `main`. - -## Using Codex (file-based prompt) -- Canonical runbook lives at `.agent-os/instructions/core/codex-runbook.md`. -- Invoke Codex with a file-based system prompt flag (example): - ```bash - codex run --prompt-file .agent-os/instructions/core/codex-runbook.md - ``` -- Aurea outputs PLAN → DIFFS → COMMANDS → TESTS → VERIFICATION → ROLLBACK; Codex applies the diffs and pushes per the runbook. - -## Known Issues / Notes -- Rotate any real API keys in `.env` before sharing. -- InfraNodus client stubs queue requests offline; integrate the real API when keys are set. -- Pinecone path is optional; with `USE_PINECONE!=1`, the system uses file memory. -- Bright Data integration is template‑based; do not enable all categories by default. - -## Soulfield OS — Docs Zone - -## TruthLens (runtime + repo gate) -All assistant and agent outputs are wrapped by TruthLens at runtime, enforcing structural clarity and non-simulation. The canonical shim lives at `truthLens.js`; the backend module is `backend/truthlens.cjs`. - -Repo hard-gate: CI (`.github/workflows/truth-kernel.yml`) blocks merges unless the Truth audit and tests pass. The audit script validates that generated artifacts either: -- contain `meta.lens.passed` with `"structure"` when JSON, or -- contain no simulation phrases in prose (code blocks are ignored). - -Run locally: +> The AI Business OS That Doesn’t Hallucinate +> Orchestrates research → strategy → execution → analytics, all gated by TruthLens. + +--- + +## ✨ What It Is +Soulfield OS is a backend-first orchestration system where: +- Aiden (Claude) turns research into specs. +- InfraNodus identifies content/market gaps. +- Bright Data Scraper fetches compliant marketplace/search data (allowlist enforced). +- Jina reranks and semantically searches across workspace docs. +- TruthLens checkpoints every step, blocking simulation, hallucinations, or invalid metrics. + +Outputs: trusted, auditable business blueprints. + +--- + +## 🚀 Quickstart + ```bash +git clone https://github.com/mrhpython/Soulfield +cd Soulfield npm ci -node backend/scripts/audit-truth.cjs -npm test + +# Research loop +sf research "eco-friendly digital planners UK" +sf specify +sf index +sf dry +sf apply --apply ``` -Policy source of truth: `workspace/knowledge/TruthLens.md`. +--- -## Codex-first workflow -We prefer plans → diffs → commands → tests → verification → rollback. Codex applies patches and pushes. See runbook: -``` -.agent-os/instructions/core/codex-runbook.md -``` +## 🛡️ Compliance +- Default jurisdiction: UK, currency GBP. +- Online-first businesses only. +- Scraping only from allowlisted templates/domains (`backend/services/scraper/config/allowlist.yaml`). +- No login-wall, PII, or policy-violating sources. +- TruthLens audit enforced in CI: `node backend/scripts/audit-truth.cjs`. + +--- -## Quick links -- Runtime lens: `truthLens.js`, `backend/truthlens.cjs` -- Audit script: `backend/scripts/audit-truth.cjs` -- Tests: `backend/tests/*.test.cjs` (Node’s built-in `node:test`) -- Ops status & commands: `workspace/docs/STATUS.md` +## 📚 Docs +- [Design Intent](workspace/docs/Soulfield%20OS%20Design%20Intent.md) +- [STATUS.md](workspace/docs/STATUS.md) +- [TruthLens Policy](workspace/knowledge/TruthLens.md) +- [TruthLens Vision](workspace/knowledge/TruthLens-Vision.md) -## License -See repository terms or your organization policy. No license header added by default. -### Local testing (Ubuntu) -- Run all unit tests: `npm test` -- Run the Truth audit only: `node backend/scripts/audit-truth.cjs` +--- -## PR Titles/Bodies — Docs Zone +## 🛠️ Development +- Node.js ≥20 +- CLI helper: `sf` (research/specify/apply/log) +- CI: TruthKernel audit + Node/Python checks +- Codex workflow: PLAN → DIFFS → COMMANDS → TESTS → VERIFICATION → ROLLBACK -- **Merge Runbook + Helper Reapply** - - Title: `docs: reapply merge runbook + helper scripts` - - Body: `Re-adds PR helper (tools/gh-pr-open-and-merge.cjs) and batch script (tools/merge-three-prs.sh), and marks the batch executable. No runtime changes.` - - Labels: `docs,tools` +--- -> Always reuse these strings when reopening or reapplying the merge runbook/PR helper. Do not invent new wording — keeps history clean and consistent. +## 📌 One-Liner +Soulfield OS = orchestration hub. +TruthLens = trust filter. +Together → reliable AI business operations.