Skip to content

CLAUDE.md

Latest commit

 

History

History
751 lines (615 loc) · 33.7 KB

CLAUDE.md

File metadata and controls

751 lines (615 loc) · 33.7 KB

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Architecture Overview

Soulfield OS is a backend-first AI orchestration system with truth verification at its core. The system routes user prompts through specialized agents, each with specific capabilities, and gates all outputs through a lens framework to prevent hallucinations.

Core Flow

User Prompt → council.js (orchestrator) → Agent Selection → LLM/Tool Execution → TruthLens → Output

Deployment Status (2025-10-30)

Current: Local Development

  • Running on development machine via npm start (localhost:8791, verified 2025-10-31)
  • All 15 agents operational with Claude API
  • Ready for Product 1-5 learning launches

Financial Dashboard (Prototype - 2025-10-31):

Cost Monitoring (2025-10-30):

Service Current Usage Monthly Cost Alert Threshold
Claude API (Anthropic) Active development ~$40/month ($10/week) Alert at $100/month
Claude Code CLI Max subscription $100/month Fixed cost
ChatGPT Pro Basic subscription $25/month Fixed cost
Supabase (pgvector) Free tier $0/month Migrate at 500MB vectors
Google Workspace Existing account $0/month No incremental cost
Bright Data (scraping) Allowlist-only ~$5/month Alert at $20/month
TOTAL Active development ~$170/month Alert at $245/month

Production Strategy (When Scaling):

  • Manual workflow with subscription models (ChatGPT Plus/Claude Pro/Z.ai at $20/month fixed cost)
  • Agents generate prompts → human executes via subscriptions → avoids API cost explosion
  • Re-automate with API when revenue covers costs

Z840 Server Deployment (When Revenue Justifies):

  • $500/month revenue → Buy second CPU (~$200)
  • $1000/month revenue → Upgrade RAM to 64GB (~$300)
  • $5000/month revenue → Add GPU, Telegram infrastructure
  • Philosophy: Make money FIRST, upgrade infrastructure with earnings
  • See: docs/Z840-DEPLOYMENT-CHECKLIST.md

Key Components

1. Entry Points

  • backend/index.cjs - HTTP server (POST /chat, GET /health) on port 8791 (verified 2025-10-31)
  • backend/council.js - Main orchestrator: handles agent routing, memory integration, TruthLens gating
  • backend/jobs.js - Command dispatcher for ! commands (!capture, !scrape, !note, etc.)

2. Agent System (15 agents defined in backend/data/agents.json)

Agent ID Name Alias Role Lens Pipeline AFS Integration
Core Agents
governor Governor @aiden Chief Orchestrator Strategy (3-lens) ⏭️ Planned
seo SEO Specialist @seo SEO Strategist Full (6-lens) ✅ Complete
marketing Marketing @marketing Marketing Strategist Full (6-lens) ✅ Complete
finance Finance @finance Financial Analyst Full (6-lens) ✅ Complete
content Content @content Content Strategist Full (6-lens) ⏭️ Planned
legal Legal @legal Legal Advisor Full (6-lens) ⏭️ Planned
visionary Visionary @visionary Business Strategist Full (6-lens) ⏭️ Planned
strategy Strategy @strategy Strategic Planner Strategy (3-lens) ⏭️ Planned
operations Operations @operations Operations Manager Strategy (3-lens) ⏭️ Planned
prompter Prompter @prompter Prompt Engineer Truth only ⏭️ Planned
Framework Agents (Cove 72-hour MVP)
builder Builder @builder MVP Developer Strategy (3-lens) ✅ Complete
distributor Distributor @distributor Distribution Strategist Strategy (3-lens) ✅ Complete
metrics Metrics @metrics Analytics & Tracking Truth only ✅ Complete
Tool Agents
jina Jina @jina Reranker Truth only N/A (tool)
scraper Scraper @scraper Web Scraper Minimal N/A (tool)

Agent Specializations:

  • @governor (@aiden) - Strategic decision-making, delegation, lens enforcement (Strategy pipeline: Rights→Causality→Truth)
  • @marketing - Campaigns, funnels, growth strategy, content calendars. 10 templates with workflow-first methodology. AFS integration complete.
  • @finance - Financial models, cost analysis, investor materials, burn rate. 10 templates with workflow-first methodology. AFS integration complete.
  • @seo - Keyword research, competitor analysis, local SEO, content strategy. AFS integration complete.
  • @builder - MVP development for Cove 72-hour cycles. Multi-agent project workspaces. AFS integration complete.
  • @distributor - Distribution strategy, channel selection, launch planning. AFS integration complete.
  • @metrics - Analytics setup, KPI tracking, performance monitoring. AFS integration complete.
  • @content - Content strategy, editorial calendars, content operations
  • @legal - Legal review, compliance, contracts, risk assessment
  • @visionary - Business strategy, idea generation, market opportunity synthesis
  • @strategy - Strategic planning, roadmaps, competitive analysis
  • @operations - Operational efficiency, process optimization, resource allocation
  • @prompter - Prompt engineering, AI system optimization, chain-of-thought design
  • @scraper - SERP data, market intelligence (Bright Data, allowlist-enforced)
  • @jina - Semantic search, document reranking

3. Lens Framework (✅ Complete with Middleware - 2025-10-20)

  • Base Lenses (6): All implemented and tested
    • TruthLens.js - Epistemic humility (unknowns, citations, speculation)
    • CausalityLens.js - Cause-effect validation (mechanisms, dependencies)
    • ContradictionLens.js - Internal consistency checks
    • ExtrapolationLens.js - Prediction validation, timeline feasibility
    • RightsLens.js - Privacy, security, ethical constraints
    • StructureLens.js - Preconditions, postconditions, operational rigor
  • Enhanced Lenses (2):
    • CausalQualityLens.js - Graph-based causal chain analysis (8/8 tests passing)
    • StructureLensV2.js - Local graph-based gap detection (✅ Fixed 2025-10-18)
  • LensOrchestrator: 7 pipeline types operational
    • Minimal (Truth only), Strategy (Rights→Causality→Truth), Full (all 6), Planning, Quality, Tool, Custom
    • Dynamic pipeline selection via local graph analysis
    • Production readiness: 13/13 tests passing (100%)
  • LensMiddleware: ✅ Enforcement layer added (2025-10-20)
    • Enforcement Modes:
      • strict: Halt on critical failures (governor, finance, legal)
      • adaptive: Auto-fix then re-validate (marketing, seo, builder)
      • soft: Log but continue (metrics, prompter)
    • Auto-Fix Capabilities:
      • Add [UNKNOWN] markers to uncited claims
      • Redact privacy violations → [REDACTED]
      • Add confidence qualifiers to predictions
      • Flag contradictions with warnings
      • Add precondition assumptions
    • Critical Lenses: Rights, Truth (halt on failure in strict/adaptive)
    • Warning Lenses: Structure, Contradiction (log but continue)
    • Configuration: LENS_ENFORCEMENT_MODE=adaptive (env var)
  • Test Coverage: 37+ test files, comprehensive validation
  • Agent Integration: @marketing, @finance, @seo use full pipeline; @governor uses strategy pipeline with strict enforcement

4. Memory System (✅ Supabase migration complete - 2025-10-15)

  • backend/services/memory/memory-supabase.cjs - Vector storage (pgvector) with AI-managed schema via MCP
  • backend/services/embedding.cjs - Provider-agnostic embedder (local/OpenAI/Anthropic/Google)
  • backend/services/docling/ - Training data ingestion (PDF/DOCX → Supabase)
    • document_processor.py - Docling-based document processing (IBM open-source)
    • ingest_training_data.py - Batch ingestion pipeline with namespace isolation
    • supabase_bridge.cjs - Python ↔ Node.js bridge for embeddings
  • memory.js (root) - File-based memory fallback (deprecated)
  • Training Data: Agents automatically recall from books/papers (marketing: Cialdini, finance: Rosenbaum, etc.)
  • Namespace strategy: soulfield:{agent}:training-{category} for isolation
  • Cost: $0/month (Supabase free tier) vs $70/month (Pinecone replaced)

5. Agent File System (✅ AFS integration complete - 2025-10-17)

  • backend/services/afs.cjs - Unified file operations for all agents with automatic Drive sync
  • backend/services/docling-service.cjs - Node.js wrapper for Python Docling pipeline
  • backend/services/inbox-processor.cjs - Continuous file monitoring with chokidar
  • backend/agents/handlers/AFS-INTEGRATION-GUIDE.md - Complete integration documentation
  • Directory: workspace/agent-workspace/{agents/,shared/,projects/,inbox/}
  • Drive Sync: Enabled by default for all file writes (OAuth2 with existing GCAL credentials)
  • Inbox Pipeline: Drop PDFs → auto-process with Docling → store vectors in Supabase
  • Project Support: Multi-agent collaboration directories for Cove framework MVPs
  • Cost: $0/month (uses existing infrastructure)

AFS Quick Reference:

const { AgentFileSystem } = require('./services/afs.cjs');
const afs = new AgentFileSystem();

// Write file (auto-syncs to Drive)
await afs.writeFile('marketing', 'campaigns/q1.md', content, { syncDrive: true });

// Read file (tries local first, falls back to Drive)
const { content } = await afs.readFile('marketing', 'campaigns/q1.md');

// Process PDF with Docling → Supabase
await afs.processAndStore(pdfPath, 'soulfield:marketing:training');

// Multi-agent project
await afs.createProject('mvp-2025-001', ['builder', 'distributor', 'metrics']);
await afs.writeProjectFile('mvp-2025-001', 'builder', 'spec.md', spec);

6. Google Workspace Integration (✅ Phase 2 Complete - 2025-10-18)

  • OAuth2: Auto-refresh tokens via existing GCAL credentials (8 scopes)
  • Services Implemented:
    • backend/services/google/calendar.cjs - List/get/create/update/delete events ✅ Validated
    • backend/services/google/docs.cjs - List/get/create/append documents ✅ Validated
    • backend/services/google/sheets.cjs - List/get/create/update ranges ✅ Validated
    • backend/services/google/gmail.cjs - List/get/send messages ✅ Validated
    • backend/services/google/drive.cjs - Upload, folder ops, file access ✅ Operational
    • backend/services/google/auth.cjs - OAuth2 client with refresh token handling ✅ Operational
  • Test Results: 10/10 validation tests passing (100%)
  • Agent Integration: Agents can create docs, schedule calendar events, send emails, update sheets
  • Cost: $0/month (uses existing Google Workspace account)
  • Next Steps: Drive watch webhooks (Phase 3), Obsidian Bridge (Phase 5)

7. Data Flow

council.js picks agent based on @prefix or defaults to @aiden
  ↓
Tool agents (jina/scraper) → direct execution via manager.cjs
  ↓
LLM agents → callClaude() with system prompt + optional memory recall
  ↓
Response → LensMiddleware (enforcement layer)
  ↓
  ├─ strict mode → HALT on critical failure → Return error to user
  ├─ adaptive mode → Auto-fix violations → Re-validate → Continue if fixed
  └─ soft mode → Log issues → Continue anyway
  ↓
Validated output → LensOrchestrator (dynamic pipeline selection)
  ↓
File outputs → AFS (automatic Drive sync + optional Docling processing)
  ↓
Google Workspace updates (Calendar events, Docs, Sheets, Gmail)
  ↓
Memory capture if non-tool agent

Development Commands

Running the System

npm start                    # Start HTTP server on :8791 (verified 2025-10-31)
npm run start:mcp            # Start MCP server
npm run start:tui            # Start terminal UI

# CLI tool (tools/sf)
sf research "topic"          # Research via Bright Data
sf specify [spec-id]         # Generate spec from research
sf index                     # Index knowledge base
sf search "query"            # SERP search (allowlist-enforced)
sf dry                       # Dry-run spec execution
sf apply --apply             # Execute approved spec
sf log                       # View run logs

Testing

npm test                                    # Run all tests (backend/tests/*.test.cjs)
node backend/tests/truth-lens.test.cjs     # Run TruthLens tests (29 cases)
node backend/tests/lens-middleware.test.cjs # Run LensMiddleware enforcement tests
node backend/scripts/audit-truth.cjs       # Run truth audit

Debugging Commands

# Enable lens debug mode (detailed validation logging)
LENS_DEBUG=true npm start

# Test lens pipeline with debug output
LENS_DEBUG=true node backend/scripts/test-lens-pipeline.cjs --pipeline minimal --text "Test content"
LENS_DEBUG=true node backend/scripts/test-lens-pipeline.cjs --pipeline full --text "Your content"

# Run all test fixtures with debug logging
LENS_DEBUG=true node backend/scripts/test-lens-pipeline.cjs

Context Tools

sf context_query "query"     # Query context spine
sf context_test              # Test context spine
sf context_summary           # Generate context summary

Training Data Ingestion (Docling + Supabase)

# Activate Python environment
source .venv-docling/bin/activate

# Process single document
python3 backend/services/docling/ingest_training_data.py \
  --file ./training-data/marketing/books/influence.pdf \
  --category marketing \
  --agent marketing

# Batch process directory
python3 backend/services/docling/ingest_training_data.py \
  --directory ./training-data/marketing/books \
  --category marketing

# Quick start guide
cat QUICKSTART-TRAINING-DATA.md

Knowledge Graph Ingestion (SQLite KG)

Current Status (2025-11-20):

  • 650 total documents indexed (610 Obsidian-v2 + 40 new files)
  • 159 entities extracted, 252 relationships mapped
  • 1.00% graph density (document-level connections)
  • All Obsidian vault files ingested and indexed
  • Analysis capability validated: Cross-document graph traversal operational
# Ingest codebase (manual, one-time only)
node backend/scripts/ingest-codebase.cjs

# Ingest daily notes (manual update, uses UPSERT)
node backend/scripts/ingest-to-knowledge-graph.cjs --file workspace/docs/Obsidian-v2/daily/2025-11-14.md

# Ingest from predefined sources
node backend/scripts/ingest-to-knowledge-graph.cjs --source daily      # All daily notes
node backend/scripts/ingest-to-knowledge-graph.cjs --source reference  # Reference docs
node backend/scripts/ingest-to-knowledge-graph.cjs --source plans      # Active plans

# Obsidian Automation Scripts (2025-11-14)
node backend/scripts/obsidian-automation/generate-topic-clusters.cjs       # Auto-discover doc clusters
node backend/scripts/obsidian-automation/identify-documentation-gaps.cjs   # MCP INSIGHTS gap detection
node backend/scripts/obsidian-automation/enrich-frontmatter.cjs            # Add graph metadata to YAML
node backend/scripts/obsidian-automation/generate-cross-reference.cjs      # Topic relationship matrix
node backend/scripts/obsidian-automation/ingest-missing-files.cjs          # Batch ingestion with backup

# Verify KG health
node backend/scripts/verify-kg-health.cjs

# Cleanup duplicates (if needed)
node backend/scripts/cleanup-kg-duplicates.cjs --dry-run
node backend/scripts/cleanup-kg-duplicates.cjs --execute

KG Update Policy:

  • Daily notes: Auto-ingested via auto-obsidian-sync.cjs after agent interactions (UPSERT, no duplicates)
  • Codebase files: Manual ingestion only via ingest-codebase.cjs (do NOT auto-ingest on every edit)
  • Reference docs/plans: Manual ingestion via ingest-to-knowledge-graph.cjs --source (UPSERT)
  • Obsidian vault: Full vault ingested (521 files). Use ingest-missing-files.cjs for incremental updates.
  • UNIQUE constraint: documents.file_path prevents duplicates at database level
  • Purpose: KG is source of truth for search, not file change history

Obsidian Graph Improvements (2025-11-14):

  • Guide: workspace/docs/Obsidian-v2/docs/guides/obsidian-graph-improvements.md
  • MCP queries: workspace/docs/Obsidian-v2/docs/guides/mcp-query-snippets.md
  • Navigation: workspace/docs/Obsidian-v2/GRAPH-NAVIGATION.md
  • Next: Metadata enhancement plan in plans/active/OBSIDIAN-METADATA-ENHANCEMENT.md

High-Risk Zones (Do Not Modify)

🔴 CRITICAL - Avoid touching until scheduled refactor:

  1. jobs.js security issue ✅ FIXED 2025-10-10

    • Command injection in !backup fixed with strict input sanitization
    • !coder commands removed (no longer exist in codebase)
    • All ! commands now use safe input validation

  2. council.js (lines 275-342) - Core orchestration logic

    • High blast radius - breaks all agents if modified
    • Tool routing with complex fallback logic
    • Add lenses as middleware, don't modify routing

  3. memory-pinecone.cjs - Vector memory operations

    • Missing namespace isolation (multi-tenancy gap)
    • Schema migration needed (Week 5)
    • Use existing APIs, don't change signatures

Safe Development Patterns

✅ Safe to create:

  • New lens modules in backend/lenses/ (zero dependencies)
  • Test files in backend/tests/
  • Documentation in workspace/docs/Obsidian/plans/
  • Memory queries using existing Pinecone APIs

✅ Safe to modify:

  • backend/data/agents.json - Add agents or update system prompts (append-only)
  • Test fixtures in backend/tests/fixtures/
  • Plan/spec files in backend/.agent-os/specs/

⚠️ Modify with caution:

  • Agent handlers (backend/agents/handlers/*.cjs) - Wait until lens framework ready
  • Adapter modules - Don't change request/response schemas

Production Readiness (✅ Validated 2025-10-18)

Lens Framework:

  • All 6 base lenses + 2 enhanced lenses complete
  • LensOrchestrator with 7 pipeline types operational
  • Production test: 13/13 passing (100%)
  • Test coverage: 36+ test files

Google Workspace:

  • All 5 services validated (Calendar, Docs, Sheets, Gmail, Drive)
  • OAuth2 auto-refresh working
  • Validation test: 10/10 passing (100%)

Agent File System:

  • 659-line implementation, 10/10 tests passing
  • Automatic Google Drive sync enabled
  • Multi-agent project workspaces operational

Memory System:

  • Supabase pgvector operational
  • Namespace isolation working
  • Training data ingestion via Docling

Next Phases:

  • Phase 3: Drive watch webhooks (automated document processing)
  • Phase 5: Obsidian Bridge (bidirectional task sync)
  • Phase 6: End-to-end workflow testing

Environment Variables

Required:

ANTHROPIC_API_KEY=           # Claude API (primary LLM)
PINECONE_API_KEY=            # Vector memory
PINECONE_INDEX=soulfield-memory
AIDEN_MODEL=claude-sonnet-4-5-20250929

Optional:

CONTEXT_SPINE=1              # Enable calendar/time context
USE_PINECONE=1               # Use vector memory (vs file-based)
NODE_ENV=production          # Disables dangerous commands
PERPLEXITY_API_KEY=          # RAG escalation (Week 2)
TAVILY_API_KEY=              # RAG escalation (Week 2)
BRIGHTDATA_TOKEN=            # Web scraping

Git Discipline

  • Branch from main: git checkout -b feat/<topic> → git push -u origin …
  • Auto-delete head branches enabled - merged PRs disappear automatically
  • Resolve conflicts by rebasing onto origin/main; avoid manual merges
  • Close legacy PRs once merge wrapper lands

Key Documentation

System Architecture:

Debugging & Development Tools:

  • claude-trace - HTTP proxy for debugging Claude Code API traffic (JSONL logs + HTML interface)
  • Claude Code Docs Map - Self-referencing documentation system (45+ official docs indexed)

Usage:

# Debug API traffic
npx @mariozechner/claude-trace --include-all-requests

# View traces
open .claude-trace/*.html

Agent System Prompt Pattern

Reference @seo agent in backend/data/agents.json for full 6-lens framework example:

  • Sequential lens processing (Truth → Causality → Contradiction → Extrapolation → Rights → Structure)
  • Explicit markers ([UNKNOWN], [HYPOTHESIS], DATA:/INTERPRETATION:/SPECULATION:)
  • Halt conditions for critical failures
  • JSON schema validation for outputs

Current gap: Other agents don't have lenses - being added Week 3-4.

Claude Code Lens Contract: See CLAUDE-LENS-CONTRACT.md for non-simulation rules and truth-first engineering constraints applied to assistant responses.

Atomic Reference System (2025-10-20)

Location: workspace/docs/Obsidian/docs/reference/ Purpose: Single source of truth for all Soulfield OS components

Structure

  • 45 reference files organized in 4 categories:
    • agents/ (16 files) - All agent references (@marketing, @finance, @seo, etc.)
    • lenses/ (10 files) - Truth validation components
    • services/ (15 files) - System services (memory, AFS, Google, etc.)
    • frameworks/ (4 files) - Architectural frameworks (Cove, Lens, AFS, Learning)
  • Master index: workspace/docs/Obsidian/docs/reference/INDEX.md

Obsidian Search

All reference files have searchable YAML frontmatter:

tag:#marketing          → All marketing-related files
type:agent              → All 16 agent references
agent:marketing         → Marketing agent specifically
category:lens           → All 10 lens references
status:active           → Active components only

Archive System

  • Deduplicated archives: 581 unique files (from 782 originals)
  • External storage: /home/michael/Documents/soulfield-archive/2025-10/
  • Archive manifest: workspace/archive-unified/index.json
  • Categories: session-summaries, integration-reports, plans, reference, concepts, other

Knowledge Extraction

  • Tool: tools/extract-knowledge.cjs
  • Status: Created, requires ripgrep installation for full functionality
  • Usage: sudo apt-get install ripgrep && node tools/extract-knowledge.cjs
  • Purpose: Mines 581 archived documents to populate reference files with AI summaries

Quick Access

  • View all components: Open workspace/docs/Obsidian/docs/reference/INDEX.md
  • Search by agent: Use Obsidian search with agent:marketing
  • Find related components: Check YAML related: field in any reference file
  • Track code locations: See YAML code_files: field for implementation paths

Auto-Sync & Rollup System (2025-10-20)

Status: ✅ Operational Purpose: Automatic knowledge capture and temporal aggregation

Auto-Sync to Obsidian

  • Service: backend/services/auto-obsidian-sync.cjs
  • Integration: Triggered automatically in backend/council.js after each agent execution
  • What it logs:
    • Agent interactions to workspace/docs/Obsidian/daily/{date}.md
    • JSONL logs to /home/michael/Documents/soulfield-archive/2025-10/session-summaries/
    • Metadata: lenses used, tools, success/failure, timestamps

Rollup System

  • Service: backend/services/rollup-service.cjs
  • Weekly summaries: Reads 7 daily notes → Creates workspace/docs/Obsidian/weekly/{year}-W{week}.md
  • Monthly rollups: Reads 4 weekly summaries → Creates workspace/docs/Obsidian/monthly/{year}-{month}.md
  • Test script: backend/scripts/test-rollup.cjs

Data Flow

Agent runs → Daily note + JSONL archive (auto)
           ↓
    Weekly rollup (manual/scheduled)
           ↓
    Monthly rollup (manual/scheduled)
           ↓
Knowledge extraction → Updates reference files

File Organization

/home/michael/Documents/soulfield-archive/2025-10/session-summaries/
├── JSONL interaction logs
├── Session summaries
└── Status reports (for knowledge extraction)

workspace/docs/Obsidian/
├── daily/          ← Auto-updated by agents
├── weekly/         ← Created by rollup service
├── monthly/        ← Created by rollup service
└── docs/
    └── reference/  ← AI-extracted knowledge (45 files)

Usage

# Knowledge extraction (reads archive → updates reference files)
node tools/extract-knowledge.cjs

# Manual rollup generation
node -e "require('./backend/services/rollup-service.cjs').generateWeeklySummary('2025-W43')"
node -e "require('./backend/services/rollup-service.cjs').generateMonthlyRollup('2025-10')"

# Test auto-sync
curl -X POST http://localhost:8791/chat -d '{"prompt": "@marketing test"}'
# Check: workspace/docs/Obsidian/daily/2025-10-20.md (auto-updated)

Soulfield Agent System (HTTP :8791)

Communication:

curl -X POST http://localhost:8791/chat -d '{"prompt": "@marketing Create campaign"}'

Routing: User prompt → backend/council.js → Agent selection → LLM/Tools → Lens validation → Output

15 Operational Agents:

  • @governor (@aiden) - Chief orchestrator, strategic planning
  • @marketing - Campaigns, funnels, growth (full 6-lens)
  • @finance - Financial models, burn rate analysis (full 6-lens)
  • @seo - SEO strategy, keyword research (full 6-lens)
  • @builder - MVP development, Cove 72-hour framework
  • @distributor - Distribution strategy, channel selection
  • @metrics - Analytics, KPI tracking
  • @content - Content strategy, editorial calendars
  • @legal - Legal review, compliance, contracts
  • @visionary - Business strategy synthesis
  • @strategy - Strategic planning, roadmaps
  • @operations - Process optimization
  • @prompter - Prompt engineering
  • @scraper - Web scraping (Bright Data)
  • @jina - Semantic search, reranking
  • @infranodus - Knowledge graphs, gap analysis

Agent Workspaces:

  • Per-agent: workspace/agent-workspace/agents/{agent-name}/
  • Multi-agent projects: workspace/agent-workspace/projects/
  • Files auto-sync to Google Drive via AFS

MCP Integration

9 MCP Servers:

  1. Soulfield KG - Knowledge graph analysis, cross-document intelligence (✅ 2025-11-20)
  2. Supabase - Database queries, agent memory management
  3. Google Workspace - Calendar, Docs, Sheets, Gmail, Drive
  4. Apify - Web scraping, data collection
  5. Playwright - Browser automation, testing
  6. Perplexity - AI research with citations
  7. Ref - Documentation search
  8. Sequential Thinking - Complex problem solving
  9. Playwright Extension - Advanced browser control

Usage: Use MCP tools proactively for database queries, calendar operations, research, and browser automation.

Knowledge Graph Analysis Capability (✅ Milestone 2025-11-20)

What Was Proven:

  • Cross-document analysis via MCP soulfield-kg server
  • Graph traversal: clusters, gaps, shared concepts detected
  • Real-time queries across 650-document knowledge graph
  • Actionable insights with file:line citations extracted automatically

MCP Soulfield-KG Tools (8 operational):

Tool Purpose Example Use Case
search Graph-completion queries with scoring "Find strategic concepts migration relationships"
getStats KG health metrics Document count, entity/relationship stats
multiHopPath Multi-step relationship traversal "Connect migration document to training data"
causalChain IF/THEN logic paths "What causes low ranking → traffic loss?"
disambiguateEntity Type-based entity filtering Distinguish "builder" (agent vs design pattern)
temporalConflict Timestamp-based resolution "Who currently owns MCP integration?"
explicitRelationships Verified-only queries "Which agents does @finance coordinate with?"
getPerformance Query performance metrics Cache hit rates, response times

Analysis Patterns Demonstrated (2025-11-20):

# Cross-document strategic analysis
mcp__soulfield_kg__search("Strategic concepts migration DSPy",
  options: {includeGraph: true, useGraphScoring: true})

# Multi-hop relationship discovery
mcp__soulfield_kg__multiHopPath("migration", "training data", maxHops: 5)

# Causal chain analysis
mcp__soulfield_kg__causalChain("training data", "agent optimization")

# Entity disambiguation
mcp__soulfield_kg__disambiguateEntity("DSPy", type: "technology")

Proven Replication Potential:

Business Use Case Input Documents Output
Strategic Planning Roadmaps, architecture docs Gap analysis, priorities
Financial Analysis Models, market data Strategic recommendations
SEO Research Competitor audits, keywords Content strategy, ranking opportunities
Website Optimization Analytics, content inventory Optimization roadmap
Agent Training Production logs, training data Continuous improvement loop

Performance Characteristics:

  • Query latency: <500ms (650 documents)
  • Graph analysis: Clusters, gaps, gateways computed in real-time
  • Cost: $0/month (local SQLite infrastructure)
  • Scalability: Proven at 650 docs, extrapolates to 10,000+

Evidence:

  • Test document 1: 2025-10-28-MIGRATION-COMPLETE.md (328 lines analyzed)
  • Test document 2: TRAINING-DATA-INVENTORY.md (701 lines analyzed)
  • Cross-references: 30+ shared concepts identified
  • Graph metrics: 29 nodes, 28 edges, 12 communities detected
  • Actionable insights: 4 strategic gaps identified with solutions

Integration Status:

  • Backend: backend/services/knowledge-graph/mcp-server.cjs (8 tools)
  • Database: workspace/data/knowledge-graph.db (650 documents)
  • Configuration: .mcp.json (soulfield-kg server enabled)
  • Skills: 13 operational skills (8 graph analysis + 5 RAG workflows)
  • Documentation: workspace/docs/Obsidian-v2/docs/reference/tools/mcp-soulfield-kg.md

Next Applications Ready:

  • Business intelligence reports (financial, market, competitive)
  • SEO content gap analysis and keyword strategy
  • Multi-agent workflow optimization
  • Strategic decision support with graph-backed evidence
  • Automated insight extraction from documentation corpus

Documentation Structure

Reference docs: workspace/docs/Obsidian/docs/reference/

  • agents/ - Agent references
  • lenses/ - Lens validation components
  • services/ - System services
  • frameworks/ - Architectural frameworks

Plans: workspace/docs/Obsidian/plans/

  • active/ - Current work
  • completed/ - Finished work
  • architecture/, integrations/, security/ - Category-specific

Temporal: workspace/docs/Obsidian/

  • daily/{YYYY-MM-DD}.md - Auto-generated daily notes
  • weekly/{YYYY-Wnn}.md - Weekly summaries
  • monthly/{YYYY-MM}.md - Monthly rollups

Root files:

  • README.md - Project overview
  • CHANGELOG.md - Version history (always update)
  • CLAUDE.md - This file (technical architecture)

Recent Major Work

Knowledge Graph Analysis Capability (2025-11-20):

  • MCP soulfield-kg server operational with 8 analysis tools
  • Cross-document intelligence validated: 650 documents, 159 entities, 252 relationships
  • Proven replication across business domains (strategic planning, SEO, finance, agent training)
  • Graph traversal: clusters, gaps, causal chains, multi-hop paths computed in real-time
  • Performance: <500ms queries, $0/month cost, scales to 10,000+ documents
  • Evidence: 2 complex documents analyzed (1,029 total lines), 30+ shared concepts identified
  • Milestone: Business-wide analysis capability proof of concept complete

RAG Workflow Skills Integration (2025-11-20):

  • 5 new skills created for RAG failure solutions (multi-hop, causal, disambiguation, temporal, verified)
  • 13 total skills operational (8 graph analysis + 5 RAG workflows)
  • Agent training data system documented (20 examples, 72 files, 100% validated)
  • Production logging system for continuous DSPy optimization
  • SQLite-only architecture confirmed (no Supabase for training data)

Multi-Agent Orchestration (2025-10-26):

  • Sequential orchestration in backend/services/orchestrator.cjs
  • 5-agent workflow tested (@governor → @finance → @seo → @marketing → @builder)
  • Integration with council.js:1505-1587

Obsidian-v2 Vault:

  • Truth-validated vault with 36 foundation files
  • Location: workspace/docs/Obsidian-v2/
  • Every claim cites code (file:line) or marked [UNKNOWN]

Decision Authority Matrix

✅ Do Without Asking:

  • Run tests after code changes
  • Update CHANGELOG.md after features
  • Use doc-writer after completion
  • Create git commits when user requests
  • Fix lint/type errors
  • Add missing file:line citations

⚠️ Ask First:

  • Modify high-risk zones (council.js:275-342 core logic)
  • Change API contracts or schemas
  • Delete files outside archive system
  • Modify production environment variables
  • Force-push to main branch

🛑 Never Do:

  • Skip lens validation in strict mode
  • Commit secrets or credentials
  • Modify .gitignore to expose sensitive files
  • Execute destructive commands without explicit request