Codex/implement hybridragservice orchestration

CHANGELOG.md

@@ -8,7 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
-- Initial release of MindMatrix plugin
+- Initial release of Obsidian RAG plugin
 - Database management features
   - Automated database setup
   - Connection testing
@@ -30,6 +30,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Database connection issues
 - Environment variable handling
 - Build process optimizations
+- Front matter tag parsing now splits comma-separated values into individual tags
 
 ## [0.1.0] - 2024-04-13
 

CLAUDE.md

@@ -1,10 +1,12 @@
-# MindMatrix Architecture
+# DEPRICATED: Obsidian RAG Architecture
+
+This Project uses OpenAi Codex AGENTS.md file for this purpose.
 
 ## Overview
 
-MindMatrix is an Obsidian plugin that enhances note-taking by synchronizing documents with a Supabase vector database for AI-powered search. The plugin provides semantic search capabilities through OpenAI embeddings while maintaining data integrity and user experience. It operates entirely within the Obsidian environment, leveraging remote services (Supabase and OpenAI) for data storage and processing.
+Obsidian RAG is an Obsidian plugin that enhances note-taking by synchronizing documents with a Supabase vector database for AI-powered search. The plugin generates embeddings through a local-first Ollama integration with automatic OpenAI fallback while maintaining data integrity and user experience. It operates entirely within the Obsidian environment, leveraging remote services (Supabase and optional OpenAI) alongside self-hosted Ollama for data storage and processing.
 
-The primary purpose of MindMatrix is to enable users to create chatbots through n8n that can query and interact with their Obsidian knowledge base. By storing document embeddings in Supabase, the plugin makes the entire vault searchable and accessible to AI-powered applications, allowing users to build custom workflows and chatbots that leverage their personal knowledge base.
+The primary purpose of Obsidian RAG is to enable users to create chatbots through n8n that can query and interact with their Obsidian knowledge base. By storing document embeddings in Supabase, the plugin makes the entire vault searchable and accessible to AI-powered applications, allowing users to build custom workflows and chatbots that leverage their personal knowledge base.
 
 The plugin implements a worker actor model with an in-memory event queue to ensure immediate response to user actions while maintaining data consistency. When a user starts typing in Obsidian, the plugin immediately detects these changes and queues them for processing, even while initialization and system startup procedures are running in the background.
 
@@ -113,7 +115,7 @@ The plugin implements a worker actor model with an in-memory event queue to ensu
   - File content reading
   - Content chunking with configurable strategies
   - Metadata extraction and normalization
-  - OpenAI embedding generation
+  - Embedding generation via Ollama with OpenAI fallback
   - Supabase database synchronization
 
 - **Synchronization**:
@@ -167,7 +169,7 @@ The plugin implements a worker actor model with an in-memory event queue to ensu
 ### 7. Settings Management
 
 - **Configuration Options**:
-  - API credentials (Supabase, OpenAI)
+  - API credentials (Supabase, Ollama, optional OpenAI)
   - Processing parameters
   - Exclusion patterns
   - Chunking strategies
@@ -216,7 +218,7 @@ The plugin implements a worker actor model with an in-memory event queue to ensu
 3. **Content Processing**:
    - File is chunked according to strategy
    - Metadata is extracted from content
-   - OpenAI API generates embeddings
+   - Embedding service requests vectors from Ollama, falling back to OpenAI when necessary
    - Chunks and embeddings are prepared for storage
 
 4. **Database Synchronization**:
@@ -283,9 +285,9 @@ The plugin implements a worker actor model with an in-memory event queue to ensu
   - Test with different configurations
 
 - **API Integration**:
-  - Test OpenAI API interactions
-  - Verify embedding generation
-  - Validate embedding quality
+  - Test Ollama endpoint interactions
+  - Test OpenAI fallback flows
+  - Verify embedding generation and dimensionality
   - Test rate limiting handling
   - Verify error recovery
 
@@ -398,7 +400,7 @@ The following checklist should be used to verify the database setup is correct b
 
 ## Project Structure
 
-The MindMatrix plugin follows a simple, flat file structure organized by function:
+The Obsidian RAG plugin follows a simple, flat file structure organized by function:
 
 - **`models/`**: Data models and type definitions for domain objects
 - **`services/`**: Business logic and core functionality
@@ -505,4 +507,4 @@ The MindMatrix plugin follows a simple, flat file structure organized by functio
 
 ## Conclusion
 
-The MindMatrix architecture provides a robust, scalable solution for synchronizing Obsidian vaults with Supabase vector databases. By implementing a worker actor model with strict event ordering and atomic processing, the plugin ensures data consistency while maintaining performance. The hash-based change detection minimizes unnecessary processing and API calls, while the event queue system ensures reliable operation even during connectivity issues.
+The Obsidian RAG architecture provides a robust, scalable solution for synchronizing Obsidian vaults with Supabase vector databases. By implementing a worker actor model with strict event ordering and atomic processing, the plugin ensures data consistency while maintaining performance. The hash-based change detection minimizes unnecessary processing and API calls, while the event queue system ensures reliable operation even during connectivity issues.

INSTALL.md

@@ -7,7 +7,8 @@
 Before you begin, ensure you have:
 - [Obsidian](https://obsidian.md/) installed
 - A [Supabase](https://supabase.com) account
-- An [OpenAI](https://platform.openai.com/) API key
+- Access to an [Ollama](https://ollama.com/) server (local or remote)
+- *(Optional)* An [OpenAI](https://platform.openai.com/) API key for fallback embeddings
 
 ### Installation Steps
 
@@ -16,14 +17,14 @@ Before you begin, ensure you have:
    #### Method 1: Through Obsidian (Recommended)
    - Open Obsidian Settings
    - Go to Community Plugins
-   - Search for "Mind Matrix"
+   - Search for "Obsidian RAG"
    - Click Install and Enable
 
    #### Method 2: Manual Installation
-   - Download the latest release from [GitHub Releases](https://github.com/yourusername/mindmatrix/releases)
+   - Download the latest release from [GitHub Releases](https://github.com/smpl-ai-automations/obsidian-rag/releases)
    - Extract the files to your vault's plugins directory:
      ```
-     .obsidian/plugins/mind-matrix/
+     .obsidian/plugins/obsidian-rag/
      ```
    - Restart Obsidian
    - Enable the plugin in Community Plugins settings
@@ -36,11 +37,14 @@ Before you begin, ensure you have:
    - Copy your Project URL
 
 3. **Configure the Plugin**
-   - Open Mind Matrix settings in Obsidian
+   - Open Obsidian RAG settings in Obsidian
    - Enter your Supabase credentials:
      - Project URL
      - Database Password
-   - Enter your OpenAI API key
+   - Configure the Embeddings section:
+     - Confirm the Ollama server URL and model (defaults to `http://localhost:11434` and `nomic-embed-text`)
+     - Enable or disable Ollama usage as needed
+     - (Optional) Provide an OpenAI API key to enable fallback embedding generation
    - Click "Initialize Database" to create the required tables
 
 4. **Start Using**
@@ -53,7 +57,7 @@ Before you begin, ensure you have:
 ### Prerequisites
 
 Before you begin, ensure you have the following installed:
-- [Node.js](https://nodejs.org/) (v16 or higher)
+- [Node.js](https://nodejs.org/) (v18 or higher)
 - [Yarn](https://yarnpkg.com/) package manager
 - [PostgreSQL](https://www.postgresql.org/) (v14 or higher)
 - [jq](https://stedolan.github.io/jq/) (for password encoding)
@@ -65,8 +69,8 @@ The inspiration for this plugin came from watching Nate Herk's YouTube video [St
 
 1. **Clone the repository**
    ```bash
-   git clone https://github.com/yourusername/mindmatrix.git
-   cd mindmatrix
+   git clone https://github.com/smpl-ai-automations/obsidian-rag.git
+   cd obsidian-rag
    ```
 
 2. **Install dependencies**
@@ -75,9 +79,9 @@ The inspiration for this plugin came from watching Nate Herk's YouTube video [St
    ```
 
 3. **Set up environment variables**
-   - Copy `.env.test` to `.env`:
+   - Copy `.env.template` to `.env`:
      ```bash
-     cp .env.test .env
+     cp .env.template .env
      ```
    - Update the `.env` file with your Supabase credentials:
      ```

Makefile

@@ -3,16 +3,23 @@
 
 # Development commands
 dev:
-	@if ! command -v node &> /dev/null; then \
-		echo "❌ Error: Node.js is not installed" && \
-		echo "Please run 'make install-node' to install Node.js" && \
-		exit 1; \
-	fi && \
-	if ! command -v yarn &> /dev/null; then \
-		echo "❌ Error: Yarn is not installed" && \
-		echo "Please run 'make install-yarn' to install Yarn" && \
-		exit 1; \
-	fi && \
+@if ! command -v node &> /dev/null; then \
+echo "❌ Error: Node.js is not installed" && \
+echo "Please run 'make install-node' to install Node.js" && \
+exit 1; \
+fi && \
+NODE_VERSION=$$(node -v | sed 's/v//') && \
+NODE_MAJOR=$${NODE_VERSION%%.*} && \
+if [ $$NODE_MAJOR -lt 18 ]; then \
+echo "❌ Error: Node.js 18+ is required (found $$NODE_VERSION)" && \
+echo "Please upgrade Node.js before running development commands" && \
+exit 1; \
+fi && \
+if ! command -v yarn &> /dev/null; then \
+echo "❌ Error: Yarn is not installed" && \
+echo "Please run 'make install-yarn' to install Yarn" && \
+exit 1; \
+fi && \
 	yarn dev
 
 # Install Node.js
@@ -73,12 +80,12 @@ install-timeout:
 
 # Check environment setup
 check-env:
-	@if [ ! -f .env ]; then \
-		echo "📝 Creating .env file from template..." && \
-		cp .env.test .env && \
-		echo "✅ .env file created. Please update the values in .env" && \
-		exit 0; \
-	fi && \
+@if [ ! -f .env ]; then \
+echo "📝 Creating .env file from template..." && \
+cp .env.template .env && \
+echo "✅ .env file created. Please update the values in .env" && \
+exit 0; \
+fi && \
 	export $$(grep -v '^#' .env | xargs) && \
 	if [ -z "$$SUPABASE_URL" ] || [ -z "$$SUPABASE_DB_PASSWORD" ]; then \
 		echo "❌ Error: SUPABASE_URL or SUPABASE_DB_PASSWORD not set in .env" && \
@@ -108,11 +115,11 @@ setup-db:
 		echo "📦 Setting up database..." && \
 		timeout 30s psql "$$DB_URL" -f sql/setup.sql && \
 		echo "✅ Database setup complete!"; \
-	else \
-		echo "❌ Error: .env file not found" && \
-		echo "Please run 'make check-env' first" && \
-		exit 1; \
-	fi
+else \
+echo "❌ Error: .env file not found" && \
+echo "Please run 'make check-env' first (uses .env.template)" && \
+exit 1; \
+fi
 
 # Initialize project and database
 init: install-node install-yarn install-postgres install-jq install-timeout check-env
@@ -179,11 +186,11 @@ test-db:
 			fi; \
 			exit 1; \
 		}; \
-	else \
-		echo "❌ Error: .env file not found"; \
-		echo "Please create a .env file based on .env.test"; \
-		exit 1; \
-	fi
+else \
+echo "❌ Error: .env file not found"; \
+echo "Please create a .env file based on .env.template"; \
+exit 1; \
+fi
 
 # Reset database command - runs reset.sql followed by setup.sql
 reset:
@@ -216,10 +223,10 @@ reset:
 			fi; \
 			exit 1; \
 		}; \
-	else \
-		echo "Error: .env file not found. Please create one based on .env.test"; \
-		exit 1; \
-	fi
+else \
+echo "Error: .env file not found. Please create one based on .env.template"; \
+exit 1; \
+fi
 
 # Release targets
 .PHONY: release-major release-minor release-patch