feat(contrib): Add SemanticCacheProcessor for semantic similarity caching

[Karanjot786]

Summary

Adds SemanticCacheProcessor, a new contrib processor that caches LLM responses based on semantic similarity using vector embeddings. Unlike exact-match caching, this approach matches queries like "What is the capital of France?" and "Tell me France's capital city" to the same cached response.

Motivation

Current caching in genai-processors uses exact hash matching, which misses cache hits for semantically equivalent queries. This causes:

• Unnecessary API calls for rephrased questions
• Higher costs in production chatbots and assistants
• Increased latency for users

Changes

New Files

• genai_processors/contrib/semantic_cache.py - Main implementation
• genai_processors/contrib/semantic_cache.md - Documentation
• genai_processors/contrib/tests/semantic_cache_test.py - Test suite (38 tests)

Modified Files

• genai_processors/contrib/README.md - Added to processor list

Features

Feature	Description
Semantic matching	Uses Gemini text-embedding-004 + cosine similarity
Configurable threshold	Default 0.90, adjustable per use case
TTL expiration	Entries expire after configurable duration
LRU eviction	Removes least-used entries when cache is full
Cache metadata	Adds hit/miss info to output part metadata
Extensible backends	VectorCacheBase ABC for custom implementations

Usage

from genai_processors.contrib import semantic_cache
from genai_processors.core import genai_model

model = genai_model.GenaiModel(api_key=API_KEY, model_name="gemini-3-flash-preview")

cached_model = semantic_cache.SemanticCacheProcessor(
    wrapped_processor=model,
    api_key=API_KEY,
    similarity_threshold=0.90,
)

# First call - cache miss
result1 = processor.apply_sync(cached_model, ["What is the capital of France?"])

# Second call - cache hit (semantically similar)
result2 = processor.apply_sync(cached_model, ["Tell me France's capital"])

[gemini-code-assist]

Summary of Changes

Hello @Karanjot786, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a powerful new "SemanticCacheProcessor" to the "genai-processors" library, fundamentally changing how LLM responses can be cached. Instead of relying on brittle exact-match caching, this new processor leverages vector embeddings and cosine similarity to identify and serve responses for semantically similar queries. This intelligent caching mechanism is designed to drastically cut down on redundant API calls, reduce operational costs, and improve response times for users interacting with LLM-powered applications, making the system more efficient and cost-effective.

Highlights

• Semantic Caching Introduction: A new "SemanticCacheProcessor" has been added to the "contrib" module, enabling caching of LLM responses based on semantic similarity rather than exact string matching.
• Embedding-based Matching: The processor utilizes vector embeddings (specifically Gemini's "text-embedding-004" model) and cosine similarity to identify semantically equivalent queries, improving cache hit rates for rephrased questions.
• Performance and Cost Optimization: This feature aims to significantly reduce unnecessary LLM API calls, lower operational costs, and decrease latency for users by serving cached responses for similar queries.
• Configurable Cache Behavior: The cache offers configurable parameters such as "similarity_threshold", Time-To-Live (TTL) for entries, and Least Recently Used (LRU) eviction policies, allowing fine-tuned control over caching behavior.
• Extensible Cache Backends: An abstract base class "VectorCacheBase" is provided, allowing developers to implement custom cache backends (e.g., Redis, FAISS) beyond the default "InMemoryVectorCache".
• PartProcessor Variant: A "SemanticCachePartProcessor" is also included, designed for caching responses of individual parts in high-concurrency scenarios.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a SemanticCacheProcessor to cache LLM responses based on semantic similarity, which is a great feature for reducing API calls and latency. The implementation is well-structured with a clear separation of concerns between the embedding client, cache storage, and the processor logic. The addition of comprehensive documentation and a thorough test suite is commendable.

My review includes a few suggestions for minor improvements, such as removing unused code, simplifying part creation for consistency, and correcting a potential typo in the documentation.

[Karanjot786]

Hi @kibergus, when you have a moment, could you please review this PR? Thanks!

[aelissee]

You have now the to_dict and from_dict method in ProcessorPart, you could use them directly. Not sure this is in the latest packaged version but we plan to release the new one very soon. So best to use those.

[Karanjot786]

Thanks for pointing this out. I'll switch to ProcessorPart.to_dict() and ProcessorPart.from_dict() directly. Will update once the new release lands.

[aelissee]

nit: use it in return statement directly, no need to compute it if norm is zero.

[Karanjot786]

Good catch. I'll move the computation into the return statement and skip it entirely when norm is zero.

[aelissee]

no need for it: ProcessorPart.to_dict should work here (works for multi-model data btw).

[Karanjot786]

I'll remove _serialize_part and use ProcessorPart.to_dict() instead. Cleaner and handles multi-modal data too.

[aelissee]

raise an exception if you have non text content?

We could generate a cache miss whenever the prompt contains non-textual data.

[Karanjot786]

Makes sense. I'll add a check: if the prompt contains non-textual parts, we generate a cache miss. No silent failures.

[aelissee]

needs to be a list if limit > 1

[Karanjot786]

I'll change the return type to list[SimilaritySearchResult] so it properly supports limit > 1.

[aelissee]

not aure you'd need this - you could handle empty input_content inside the embed method - raising an exception, and the following block would apply.

[Karanjot786]

I'll move the empty input handling into the embed method itself and raise an exception there. Removes the need for the early return block.

[aelissee]

you could put this into a separate thread asyncio.to_thread() as it might take a while, then the asyncio loop does not block on it.

[Karanjot786]

I'll wrap the cleanup loop in asyncio.to_thread() so the event loop stays unblocked during eviction.

[aelissee]

note: only works for turn based processor, not for realtime ones.

[Karanjot786]

Will add a clear note in the docstring: this processor works for turn-based use cases only, not realtime ones.

[aelissee]

Hi, thanks for the proposal: I'd let Kibergus check it as well but first quick pass at it.

[Karanjot786]

@aelissee @kibergus Applied all the review feedback. Here is what changed:

Serialization:

• Replaced custom _serialize_part with ProcessorPart.to_dict() for storage
• Used ProcessorPart.from_dict(data=part_dict) for deserialization

Error handling:

• embed() now raises ValueError for empty content and non-text parts
• This triggers a cache miss in call(), which falls through to the wrapped processor
• Removed the separate empty-input check from call()

API changes:

• find_similar() returns list[SimilaritySearchResult] instead of SimilaritySearchResult | None
• Supports limit parameter for multiple results

Performance:

• Inlined dot product in cosine_similarity, returns 0.0 when norms are zero
• Wrapped _evict_expired in asyncio.to_thread() to avoid blocking the event loop

Documentation:

• Added turn-based-only limitation to class docstring and docs
• Updated custom cache backend example with new find_similar signature
• Expanded limitations section for non-text content handling
• Removed _serialize_part from API reference

Tests:

• Updated all assertions for list return type
• Added tests for find_similar with limit > 1
• Added test for non-text content bypass
• Updated get_response_parts test to use to_dict/from_dict format
• Fixed concurrency test race condition
• All 41 tests pass with 0 warnings

[kibergus]

Karanjot, thank you for the contribution, but I think it would be better, more maintainable if it lives in a repository that you own. We can add a link to that repository to a list in contrib/README.md https://github.com/google-gemini/genai-processors/tree/main/genai_processors/contrib#readme similar to how there are links to mbeacom's contrinutions.

If it lives in our repository, we ourselves won't be able to provide much maintenance/support to it. And you will need to go through code review from us to make any change. This won't be conveninet for either of us.

SemanticCacheProcessor explores an interesting idea, but is application-specific. It needs careful tuning to avoid quality degradation due to false positives. E.g. "Tell me the 1682 France's capital city" has a different answer from "Tell me France's capital city". It is hard to find a universal threshold. Another problem is that computing embedding is not free. If users of the system don't ask similar questions really often, it can easily increase the cost of running.

The cache we currently have targets a somewhat different problem: allow resuming a previoiusly interrupted operation. It is handy when developing a multi-stage pipeline or if the pipeline takes a long time to complete. And in this case exact match can easily be achieved.