Deduplicate near-identical content at retain time

[mvessair-hive]

Problem

When an LLM agent recalls memories and then retains a new interaction in the same turn, the recalled context often gets re-retained as new facts. Over multiple turns this creates exponential duplication — we observed a single fact ("X was out with Y") duplicated 31 times across world and observation types.

This affects any agent that follows a recall → respond → retain pattern, which is the standard usage loop for Hindsight-backed agents.

Proposed Solution

At retain time, before creating new memory units, check embedding similarity of the incoming content against existing memories in the same bank. If similarity exceeds a threshold (e.g. >0.9 cosine), skip or merge instead of creating a new fact.

This could be:

• A configurable bank-level setting (dedup_threshold: 0.9)
• Opt-in via a retain parameter (deduplicate: true)
• Always-on with a high threshold to avoid false positives

Impact

Without dedup, banks accumulate redundant copies that degrade recall quality (redundant results crowd out diverse facts) and waste storage/embedding compute.

[nicoloboschi]

if you're adding turns to a conversation, you should assign a fixed document id so Hindisght can do a proper delta update and no duplicates are guaranteed.
https://hindsight.vectorize.io/best-practices#the-document_id-field

[kaiisfree]

Technical analysis: retain-time dedup — code path trace and implementation proposal

I traced through the retain pipeline to identify where dedup could be inserted and what infrastructure already exists.

Current retain code path

The retain flow is: MemoryEngine.retain_batch_async() → orchestrator.retain_batch() → _extract_and_embed() → _streaming_retain_batch() / _try_delta_retain().

Within _extract_and_embed(), the pipeline does:

1. Fact extraction via LLM call that decomposes content into atomic facts
2. Embedding generation — computes embeddings for all extracted facts
3. Returns ProcessedFact objects, each with its .embedding vector

Then in Phase 1 (_pre_resolve_phase1()), the system already runs ANN search against existing memory_units via compute_semantic_links_ann() in link_utils.py. This function creates a temp table of seed embeddings, runs CROSS JOIN LATERAL against the HNSW index on memory_units.embedding with <=> (cosine distance), and filters by similarity >= 0.7.

Key finding: cosine similarity against the full bank is already computed for every incoming fact during retain, as part of semantic link creation. The infrastructure for dedup detection is already running — it just doesn't treat very-high-similarity matches as duplicates.

Why document_id doesn't fully solve this

The delta retain path uses chunk-level SHA256 hashes — content-hash dedup at the chunk level. Three gaps remain:

1. Cross-document dedup: Each retain call with no explicit document_id gets a fresh uuid4(). The recall→respond→retain loop generates a new call per turn.
2. Semantic equivalence vs textual identity: "Alice works at Google" and "Alice is employed at Google" have different SHA256 hashes but are semantically identical. Only embedding similarity catches these.
3. Cross-conversation: Agent using bank "user123" in conversation A recalls and re-retains a fact. Conversation B does the same independently. No shared document_id.

Where dedup should be inserted

The ideal insertion point is after embedding generation but before insert_facts_batch() — specifically in _streaming_retain_batch(), between the _extract_and_embed() call and the Phase 1 pipeline.

After this line:

extracted_facts, processed_facts, chunks, usage = await _extract_and_embed(...)

...and before Phase 1, insert a dedup filter that:

1. Takes the processed_facts list (each has .embedding)
2. Runs ANN query against existing memory_units for the same bank (same temp-table + LATERAL pattern as compute_semantic_links_ann(), but with top_k=1 and threshold=dedup_threshold)
3. Filters out any fact where nearest existing neighbor exceeds threshold

Proposed implementation

Bank-level config: retain_dedup_threshold

retain_dedup_threshold: float | None = hierarchical(None)  # None = disabled, 0.90-0.99 recommended

Fits naturally into the existing config hierarchy (env vars → tenant → bank overrides via ConfigResolver). Being hierarchical means it can be set per-bank via the banks.config JSONB column, matching how retain_extraction_mode and retain_chunk_size already work.

Dedup filter

New function, either in engine/retain/embedding_processing.py or a new engine/retain/dedup.py:

async def filter_duplicates(
    pool, bank_id: str, processed_facts: list[ProcessedFact],
    threshold: float, log_buffer: list[str]
) -> list[ProcessedFact]:
    if not processed_facts or threshold is None:
        return processed_facts
    # Reuse temp-table + LATERAL pattern from compute_semantic_links_ann
    # but with top_k=1 and the dedup threshold
    # ...
    novel = [f for i, f in enumerate(processed_facts) if i not in duplicate_indices]
    log_buffer.append(f"  Dedup: {len(processed_facts) - len(novel)} filtered at threshold={threshold}")
    return novel

Cost analysis

The ANN query for dedup is cheap relative to what retain already does — compute_semantic_links_ann() already runs with top_k=50 and ef_search=60 for semantic links. Dedup with top_k=1 and ef_search=40 is roughly 30x cheaper per probe. For 100 facts, this adds one temp table + one LATERAL query — milliseconds on a warmed HNSW index.

The expensive part is the LLM fact extraction, which happens before dedup. A future optimization could do dedup on raw content before LLM extraction, but post-extraction catches paraphrased duplicates more reliably.

Open questions

• False-positive rate at various thresholds. 0.95 is safe for exact re-retention but misses paraphrases. 0.90 catches more but risks merging legitimately similar-but-distinct facts. Empirical testing needed.
• Whether fact_type filtering should apply — the ANN search already splits by fact_type to use per-type HNSW indexes.
• HNSW warm-up time on cold banks (the existing ANN code sets a 300s timeout, suggesting large banks can be slow).

Happy to draft a PR if this direction looks right.

[nicoloboschi]

If those are different documents, then just recall observations only

[kaiisfree]

That makes sense — scoping dedup to the observation layer is a cleaner architectural fit since observations are where the high-volume, high-overlap retention happens. Facts and documents have stronger identity semantics that document_id already handles well. One question: does the existing ANN infrastructure in the retain pipeline (the HNSW index + LATERAL query pattern used by compute_semantic_links_ann) already support observation-level similarity queries, or would that need to be added as a separate index/query path? If it's already there, the dedup filter becomes a thin wrapper around existing machinery.