InsightDocs

An AI-driven agent architecture system for transforming unstructured documents into operational intelligence through multi-agent collaboration and RAG (Retrieval-Augmented Generation).

🎯 Overview

InsightDocs is a production-ready platform that combines multi-agent AI architecture with RAG capabilities to process, analyze, and query documents intelligently. Built with modern microservices patterns, it demonstrates best practices for building scalable, maintainable AI applications.

Key Capabilities

📄 Document Intelligence: Upload and process PDFs, Word documents, and text files
🔐 Secure BYOK Architecture: Bring Your Own Key with AES-256 encryption and tenant isolation
🤖 Multi-Agent System: Coordinated AI agents working together on complex workflows
🔍 Semantic Search: RAG-powered queries with vector similarity search
⚡ Async Processing: Background task processing with Celery workers
🎨 REST API: FastAPI-based high-performance API endpoints
🐳 Docker Ready: Complete containerized deployment stack

🌟 Features

Multi-Agent Architecture

The system employs four specialized agents coordinated by an orchestrator:

Orchestrator Agent: Central workflow coordinator managing all agents
Data Agent: Handles document ingestion, storage, and transformation
Analysis Agent: Generates embeddings, summaries, and entity extraction
Planning Agent: Provides workflow planning and decision support

RAG (Retrieval-Augmented Generation)

Document chunking and intelligent segmentation
Vector embedding generation with sentence transformers
Milvus-powered similarity search (Hybrid Dense + Sparse)
Context-aware response generation using Gemini
Source citation and attribution

Production Infrastructure

FastAPI: High-performance async API framework
PostgreSQL: Reliable metadata and document storage
Redis: Message queuing, task broker, and caching
S3/MinIO: Scalable object storage for files
Celery: Distributed async task processing
Docker Compose: Simple deployment orchestration

📊 Architecture

┌─────────────────────────────────────────────────────────────┐
│                        Presentation Layer                    │
│  CLI Tool  │  FastAPI REST API  │  API Docs (Swagger)      │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                      Business Logic Layer                    │
│  Orchestrator → Data Agent → Analysis Agent → Planning Agent│
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                         Data Layer                           │
│  PostgreSQL (Metadata) ↔ Milvus (Vectors) ↔ Redis (Queue)  │
└─────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────┐
│                       Storage Layer                          │
│        S3/MinIO (Files) ↔ Local Storage (Temp)            │
└─────────────────────────────────────────────────────────────┘

Workflow Example: Document Processing

User → API → Document Record (PostgreSQL)
           → Celery Task → Orchestrator Agent
                         → Data Agent: Store file in S3, parse content
                         → Data Agent: Chunk text into segments
                         → Analysis Agent: Generate embeddings
                         → Data Agent: Store in Milvus
           → Task Complete → Document ready for querying

🚀 Quick Start

Prerequisites

Python 3.11 or higher
Docker and Docker Compose
Gemini API key

Installation

Clone the repository

git clone https://github.com/HarshilMaks/InsightDocs.git
cd InsightDocs

Configure environment

cp .env.example .env
# Edit .env and add your GEMINI_API_KEY

Start with Docker Compose (Recommended)

docker-compose up -d

Services will be available at:

API: http://localhost:8000
API Documentation: http://localhost:8000/docs
MinIO Console: http://localhost:9001 (credentials: minioadmin/minioadmin)

Alternative: Manual Setup

# Create virtual environment (if not exists)
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
uv pip install -r requirements.txt

# Start services (PostgreSQL, Redis, MinIO separately)

# Initialize database
python -c "from backend.models import Base, engine; Base.metadata.create_all(bind=engine)"

# Terminal 1: Start API
uvicorn insightdocs.api.main:app --reload

# Terminal 2: Start Celery worker
celery -A insightdocs.workers.celery_app worker --loglevel=info

💡 Usage Examples

Using the CLI

# Check system health
python cli.py health

# Upload a document
python cli.py upload document.pdf

# Query documents
python cli.py query "What are the key findings?"

# List all documents
python cli.py list-documents

# Check task status
python cli.py status <task-id>

Using the REST API

Upload a document:

curl -X POST "http://localhost:8000/documents/upload" \
  -F "file=@document.pdf"

Query documents:

curl -X POST "http://localhost:8000/query/" \
  -H "Content-Type: application/json" \
  -d '{"query": "What is this about?", "top_k": 5}'

List documents:

curl "http://localhost:8000/documents/"

Using Python

import requests

# Upload document
with open('document.pdf', 'rb') as f:
    response = requests.post(
        'http://localhost:8000/documents/upload',
        files={'file': f}
    )
    result = response.json()
    print(f"Document ID: {result['document_id']}")
    print(f"Task ID: {result['task_id']}")

# Query documents
response = requests.post(
    'http://localhost:8000/query/',
    json={'query': 'What are the main topics?', 'top_k': 5}
)
answer = response.json()
print(f"Answer: {answer['answer']}")
print(f"Sources: {len(answer['sources'])} chunks retrieved")

📦 Project Structure

InsightDocs/
├── insightdocs/           # Main application package
│   ├── agents/            # Multi-agent system
│   │   ├── orchestrator.py    # Central coordinator
│   │   ├── data_agent.py      # Data operations
│   │   ├── analysis_agent.py  # Analysis & embeddings
│   │   └── planning_agent.py  # Workflow planning
│   ├── api/               # FastAPI endpoints
│   │   ├── main.py            # Application entry
│   │   ├── documents.py       # Document management
│   │   ├── query.py           # RAG query endpoint
│   │   ├── tasks.py           # Task monitoring
│   │   └── schemas.py         # Pydantic models
│   ├── core/              # Core framework
│   │   ├── agent.py           # Base agent class
│   │   └── message_queue.py   # Message passing
│   ├── models/            # Database models
│   │   ├── schemas.py         # SQLAlchemy models
│   │   └── database.py        # DB connection
│   ├── utils/             # Utilities
│   │   ├── document_processor.py  # Document parsing
│   │   ├── embeddings.py          # Vector operations
│   │   └── llm_client.py          # Gemini integration
│   ├── workers/           # Celery workers
│   │   ├── celery_app.py      # Celery config
│   │   └── tasks.py           # Background tasks
│   ├── storage/           # Storage layer
│   │   └── file_storage.py    # S3/MinIO integration
│   └── config/            # Configuration
│       └── settings.py        # App settings
├── tests/                 # Test suite
├── docs/                  # Documentation
├── cli.py                 # Command-line interface
├── docker-compose.yml     # Docker orchestration
├── Dockerfile             # Container definition
├── Makefile               # Development commands
└── requirements.txt       # Dependencies

🧪 Testing

# Run all tests
pytest tests/

# Run with coverage
pytest --cov=insightdocs --cov-report=html

# Run specific test module
pytest tests/test_core_agent.py -v

🛠️ Development

Using Make Commands

make help           # Show all available commands
make install        # Install production dependencies
make install-dev    # Install development dependencies
make test           # Run test suite
make test-cov       # Run tests with coverage
make clean          # Clean cache and temp files
make run-api        # Start API server
make run-worker     # Start Celery worker
make docker-up      # Start all services
make docker-down    # Stop all services
make docker-logs    # View service logs

Development Workflow

Create a feature branch
Make changes and test locally
Run tests: make test
Clean up: make clean
Submit pull request

📚 Documentation

Quick Start Guide: Getting started with InsightDocs
Architecture Guide: Deep dive into system design
API Reference: Complete API endpoint documentation
Development Guide: Contributing and development workflow
System Overview: Detailed component breakdown

⚙️ Configuration

Environment variables (.env file):

# Database
DATABASE_URL=postgresql://insightdocs:insightdocs@localhost:5432/insightdocs

# Redis
REDIS_URL=redis://localhost:6379/0
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/1

# Gemini
GEMINI_API_KEY=sk-your-api-key-here

# Storage (S3/MinIO)
S3_ENDPOINT=http://localhost:9000
AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=minioadmin
S3_BUCKET_NAME=insightdocs

# Application
APP_ENV=development
LOG_LEVEL=INFO
SECRET_KEY=your-secret-key-here

# Vector Database
VECTOR_DIMENSION=384

🎯 Use Cases

Document Intelligence Platform: Build Q&A systems over document collections
Knowledge Base Management: Create searchable organizational knowledge repositories
Research Assistant: Query and analyze across multiple research documents
Content Analysis: Extract entities, generate summaries, analyze document content
Workflow Automation: Coordinate complex multi-step AI-powered workflows

🔒 Security Features

BYOK (Bring Your Own Key): User keys stored with AES-256 encryption
Tenant Isolation: Strict DB and Vector DB isolation per user
Rate Limiting: Authenticated user-based rate limiting
Input Validation: Pydantic validation on all endpoints
SQL Injection Prevention: SQLAlchemy ORM usage
Secrets Management: Environment-based configuration

📈 Performance & Scalability

Async I/O: Non-blocking operations throughout the stack
Connection Pooling: Efficient database connection management
Batch Processing: Optimized embedding generation
Horizontal Scaling: Scale API servers and workers independently
Caching: Redis-based caching for frequently accessed data
Vector Search: FAISS for fast similarity search at scale

🛠️ Technology Stack

Category	Technology
Language	Python 3.11+
API Framework	FastAPI
Task Queue	Celery
Database	PostgreSQL
Cache/Queue	Redis
Vector DB	Milvus
Storage	S3/MinIO
LLM	Gemini GPT
Embeddings	Sentence Transformers
Containerization	Docker, Docker Compose

🤝 Contributing

Contributions are welcome! Please follow these steps:

Fork the repository
Create a feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add some AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

Please ensure your code:

Follows the existing code style
Includes appropriate tests
Updates documentation as needed
Passes all tests (make test)

🐛 Troubleshooting

Services won't start

# Check if ports are in use
lsof -i :8000  # API
lsof -i :5432  # PostgreSQL
lsof -i :6379  # Redis
lsof -i :9000  # MinIO

# Restart services
docker-compose down
docker-compose up -d

Database connection errors

# Check PostgreSQL
docker-compose ps postgres
docker-compose logs postgres

# Recreate database
docker-compose down -v
docker-compose up -d

Worker not processing tasks

# Check worker logs
docker-compose logs worker

# Check Redis
docker-compose ps redis

📊 Project Statistics

29 Python modules implementing the complete system
2,271 lines of code in the main application
200+ lines of test coverage
4 specialized agents working in coordination
3 API routers (documents, query, tasks)
5 documentation files covering all aspects

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

🙏 Acknowledgments

Built with modern Python frameworks and best practices
Inspired by agent-based architectures and microservices patterns
Demonstrates practical implementation of RAG systems

📞 Support

Issues: GitHub Issues
Documentation: See the docs/ directory
Examples: Check out cli.py for usage examples

Built with ❤️ using Python, FastAPI, Celery, PostgreSQL, Redis, Milvus, and Gemini

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

InsightDocs

🎯 Overview

Key Capabilities

🌟 Features

Multi-Agent Architecture

RAG (Retrieval-Augmented Generation)

Production Infrastructure

📊 Architecture

Workflow Example: Document Processing

🚀 Quick Start

Prerequisites

Installation

Alternative: Manual Setup

💡 Usage Examples

Using the CLI

Using the REST API

Using Python

📦 Project Structure

🧪 Testing

🛠️ Development

Using Make Commands

Development Workflow

📚 Documentation

⚙️ Configuration

🎯 Use Cases

🔒 Security Features

📈 Performance & Scalability

🛠️ Technology Stack

🤝 Contributing

🐛 Troubleshooting

Services won't start

Database connection errors

Worker not processing tasks

📊 Project Statistics

📄 License

🙏 Acknowledgments

📞 Support

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

InsightDocs

🎯 Overview

Key Capabilities

🌟 Features

Multi-Agent Architecture

RAG (Retrieval-Augmented Generation)

Production Infrastructure

📊 Architecture

Workflow Example: Document Processing

🚀 Quick Start

Prerequisites

Installation

Alternative: Manual Setup

💡 Usage Examples

Using the CLI

Using the REST API

Using Python

📦 Project Structure

🧪 Testing

🛠️ Development

Using Make Commands

Development Workflow

📚 Documentation

⚙️ Configuration

🎯 Use Cases

🔒 Security Features

📈 Performance & Scalability

🛠️ Technology Stack

🤝 Contributing

🐛 Troubleshooting

Services won't start

Database connection errors

Worker not processing tasks

📊 Project Statistics

📄 License

🙏 Acknowledgments

📞 Support