The Scraper

An interactive, AI-driven research assistant that researches, filters, and summarizes topics.

How it Works

➤ The assistant uses a local AI model to generate search queries, browse websites, and extract relevant content. You interactively select sources and output format before the AI creates a live summary.

Features

⚙ Interactive Control: Source & format selection. 🧠 AI-Driven: Research & summarization by LLM. 💾 Intelligent Caching: Accelerates repeated searches. ⚡ Live Summary: Real-time output in the terminal. 📊 Progress Indicators: Visual feedback during long operations. 🛡 Robust Ollama Communication: Automatic retries. 🔍 Strict Content Filtering: By domain, language, length. 🔧 Easy Configuration & Validation: config.json is checked. 📦 Fully Automatic Setup: Dependencies & Ollama models. 🏠 100% Local: Full data control.

Prerequisites

🐍 Python 3.x 🐳 Ollama: A running local Ollama server.

Installation & Usage

1. Download Files: scraper.py, requirements.txt, config.json in the same directory.
2. Run Script:

   python scraper.py

3. First Start: config.json is created, Python packages installed, Ollama model checked/offered.
4. Interactive Process: Enter topic, select sources, choose format, observe summary.

The final result is saved in output.txt.

Configuration (config.json)

Adjust the assistant's behavior via the config.json file. Important settings include Ollama details, search parameters, filters, and caching options.