feat: pluggable embedding providers (OpenAI + Gemini)#206
Open
aloysiusmartis wants to merge 1 commit intogarrytan:masterfrom
Open
feat: pluggable embedding providers (OpenAI + Gemini)#206aloysiusmartis wants to merge 1 commit intogarrytan:masterfrom
aloysiusmartis wants to merge 1 commit intogarrytan:masterfrom
Conversation
Adds a provider-agnostic EmbeddingProvider interface so gbrain can use Gemini (text-embedding-004/gemini-embedding-001) instead of OpenAI, selected via GBRAIN_EMBEDDING_PROVIDER env var. The public embed/embedBatch API in embedding.ts is unchanged — callers see no diff. Architecture: - src/core/embedding-provider.ts — EmbeddingProvider interface, factory (getActiveProvider), isEmbeddingAvailable(), resetActiveProvider() - src/core/providers/openai-embedder.ts — OpenAI impl extracted from embedding.ts - src/core/providers/gemini-embedder.ts — Gemini impl with Matryoshka dims - src/core/providers/retry-utils.ts — shared exponentialDelay + sleep Critical fix: operations.ts put_page had hardcoded !process.env.OPENAI_API_KEY, so Gemini users got silent no-embed on every import. Replaced with isEmbeddingAvailable() which checks whichever provider is active. New command: gbrain migrate --provider openai|gemini [--dimensions N] - ALTER TABLE (only when dims change) - Re-embeds all chunks with the new provider - Updates config table + config.json - Remote guard: CLI-only, cannot be called via MCP Schema: getPGLiteSchema(dims, model) replaces hardcoded vector(1536) in PGLite DDL so new Gemini brains get vector(768) from init. Config: GBrainConfig gains embedding_provider + embedding_dimensions; loadConfig() propagates them to env on startup (does not override if already set). Init: gbrain init --provider gemini [--dimensions N] wires provider at brain creation time. Usage: GBRAIN_EMBEDDING_PROVIDER=gemini gbrain init # Gemini brain, 768 dims gbrain migrate --provider gemini # migrate existing brain gbrain migrate --provider openai # migrate back Relates to: upstream PR garrytan#197 (voyage embedding) — same territory but this approach uses an interface/factory pattern that supports N providers without modifying the call sites each time. Co-authored-by: Al's bot <aloysiusmartis@users.noreply.github.com>
dcad9d4 to
2445e3f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
EmbeddingProviderinterface so gbrain can use Gemini (gemini-embedding-001, 1–3072 Matryoshka dims) or OpenAI (text-embedding-3-large, 1536 dims) interchangeablyembed/embedBatchAPI inembedding.tsis unchanged — existing call sites see no diffoperations.ts put_pagehad!process.env.OPENAI_API_KEYhardcoded, so Gemini users got no embeddings on every page importArchitecture
getActiveProvider()readsGBRAIN_EMBEDDING_PROVIDERenv and returns the right singleton.isEmbeddingAvailable()replaces all!process.env.OPENAI_API_KEYchecks so hybrid search and page ingestion work correctly regardless of which provider is active.New command:
gbrain migrate --providerconfig_table+~/.gbrain/config.jsonInit-time provider selection
Schema
getPGLiteSchema(dims, model)replaces hardcodedvector(1536)in PGLite DDL sogbrain init --provider geminicreates avector(768)schema from the start.Config persistence
GBrainConfiggainsembedding_providerandembedding_dimensions.loadConfig()propagates them to env vars at startup so subsequent sessions use the same provider without repeating the flag.Tests
test/embedding-provider.test.ts— 22 unit + 3 live (skipped without API key): factory, fallback, unknown provider error, boundary dimstest/pglite-schema-provider.test.ts— 6 tests forgetPGLiteSchema()substitutionstest/config-embedding-provider.test.ts— 4 tests for env-var propagation (no-override behavior)test/migrate-provider-args.test.ts— 8 tests for dims-change logic and API key guardAll 2627 unit tests pass (
bun test, 0 fail).Relation to PR #197
This overlaps with trymhaak's voyage embedding PR (#197). The approach here uses an interface/factory pattern (
EmbeddingProvider) so adding future providers (voyage, cohere, local) is additive — no call site changes. The factory is a singlegetActiveProvider()call; allembed/embedBatchcallers go throughembedding.tsunchanged.If #197 lands first, this PR could subsume it by adding a
VoyageEmbeddertosrc/core/providers/. Happy to coordinate.Checklist
embed,embedBatch,EMBEDDING_MODEL,EMBEDDING_DIMENSIONSinembedding.ts)// FORK:comments in this diffCOORDINATION.mdand fork-only scripts not includedmigrate --provider(MCP callers get a clear error)--dimensionsarg parsing