Personal Knowledge Base

A RAG-based personal knowledge base using natural language for both input and querying, built with Go.

Technology Stack

• Vector Database: Milvus
• Metadata Storage: Redis
• Embedding Generation: VoyageAI
• Reranking: VoyageAI Rerank
• LLM: Anthropic Claude Sonnet
• Email Service: Mailjet (interface-based design)
• Backend Language: Go

Architecture Overview

The personal knowledge base consists of these major components:

1. Document Processing Pipeline: Handles incoming natural language text
2. Storage System: Manages both vector embeddings and metadata
3. Query Processing System: Processes natural language queries with reranking for improved accuracy
4. Response Generation System: Creates natural language responses

Setup & Installation

Prerequisites

• Docker and Docker Compose
• Go 1.21 or later
• VoyageAI API key
• Anthropic Claude API key

Environment Configuration

1. Copy the example environment file and fill in your API keys:

cp exmample.env .env

2. Edit .env with your VoyageAI and Anthropic API keys

Starting all services

Start all services using Docker Compose:

docker-compose up -d

Using the Web UI

Navigate to localhost:3000 to interact with hippocamp using the web UI. Here you can add new documents and ask questions using natural lanugage queries.

CLI API Usage

Adding a Document

curl -X POST http://localhost:8080/api/documents \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Example Document",
    "content": "This is the content of the document that will be processed and stored in the knowledge base."
  }'

Listing Documents

curl http://localhost:8080/api/documents

Getting a Document by ID

curl http://localhost:8080/api/documents/{document_id}

Querying the Knowledge Base

curl -X POST http://localhost:8080/api/query \
  -H "Content-Type: application/json" \
  -d '{
    "text": "What information do you have about example topics?"
  }'

Deleting a Document

curl -X DELETE http://localhost:8080/api/documents/{document_id}

Reranking Configuration

The system uses VoyageAI's reranking models to improve retrieval quality. This is configured through the following environment variables:

• RERANKER_ENABLED: Set to true to enable reranking (default: true)
• RERANKER_MODEL: The reranking model to use (default: rerank-2). Available options:
  • rerank-2: Best quality, 16K token context
  • rerank-2-lite: Good quality with faster speed, 8K token context
  • rerank-1: Legacy model (not recommended)
  • rerank-lite-1: Legacy model (not recommended)
• RERANKER_TOP_K: Number of top results to return after reranking (default: 10)

Reranking improves retrieval quality by using a more sophisticated cross-encoder model that considers both the query and document together for relevance assessment.

Architecture Details

Document Processing

When documents are added, they are:

1. Split into chunks with configurable size and overlap
2. Embedded using VoyageAI
3. Stored in Milvus for vector search
4. Metadata is stored in Redis

Query Processing

When queries are processed:

1. The query is embedded using the same VoyageAI model
2. Similar chunks are retrieved from Milvus
3. Chunks are used as context for Claude to generate a response
4. The response is returned with citations to the source material

Configuration

The application is configured via environment variables:

• VOYAGEAI_API_KEY: API key for VoyageAI embedding service
• ANTHROPIC_API_KEY: API key for Anthropic Claude
• Milvus and Redis connection settings
• Chunking parameters
• Server port and logging level

Development

Running Tests

go test ./...

Code Structure

• config/: Configuration loading from environment
• internal/api/: HTTP server and API endpoints
• internal/model/: Data models
• internal/processor/: Document and query processing logic
• internal/storage/: Vector and metadata storage interfaces

License

MIT License