feat(context): enforce sensitivity policy — drop/redact restricted items from prompts

.github/copilot-instructions.md

@@ -6,7 +6,7 @@ This is contextweaver, a Python library for dynamic context management for AI ag
 - src/contextweaver/ contains the library
 - Two engines: Context Engine (phase-specific context building) and Routing Engine (bounded-choice tool routing)
 - All stores (EventLog, ArtifactStore, EpisodicStore, FactStore) are protocol-based with InMemory defaults
-- Context pipeline: generate_candidates → score_candidates → deduplicate_candidates → select_and_pack → render_context
+- Context pipeline: generate_candidates → sensitivity_filter → apply_firewall → score_candidates → deduplicate_candidates → select_and_pack → render_context
 - Routing pipeline: TreeBuilder → ChoiceGraph → Router (beam search) → ChoiceCards
 
 ## Conventions
@@ -30,3 +30,4 @@ This is contextweaver, a Python library for dynamic context management for AI ag
 - ContextPack (rendered prompt + stats + BuildStats)
 - ChoiceCard (LLM-friendly compact card, never includes full schemas)
 - ChoiceGraph (bounded DAG, serializable, validated on load)
+- MaskRedactionHook (built-in redaction hook for sensitivity enforcement)

AGENTS.md

@@ -18,7 +18,7 @@ It provides two integrated engines:
 | `src/contextweaver/serde.py` | Serialisation helpers for to_dict/from_dict patterns |
 | `src/contextweaver/store/` | InMemoryArtifactStore, InMemoryEventLog, InMemoryEpisodicStore, InMemoryFactStore |
 | `src/contextweaver/summarize/` | SummarizationRule, RuleEngine, extract_facts() |
-| `src/contextweaver/context/` | Full context pipeline: candidates → scoring → dedup → selection → firewall → prompt |
+| `src/contextweaver/context/` | Full context pipeline: candidates → sensitivity filter → firewall → scoring → dedup → selection → prompt |
 | `src/contextweaver/routing/` | Catalog, ChoiceGraph, TreeBuilder, Router (beam search), cards renderer |
 | `src/contextweaver/adapters/` | MCP and A2A protocol adapters |
 | `src/contextweaver/__main__.py` | CLI: 7 subcommands (demo, build, route, print-tree, init, ingest, replay) |

CHANGELOG.md

@@ -8,6 +8,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- Sensitivity enforcement in context pipeline: items at or above `ContextPolicy.sensitivity_floor` are dropped or redacted
+- `ContextItem.sensitivity` field (default: `Sensitivity.public`)
+- `ContextPolicy.sensitivity_action` field (`"drop"` or `"redact"`)
+- `MaskRedactionHook` — built-in redaction hook replacing text with `[REDACTED: {sensitivity}]`
+- `apply_sensitivity_filter()` function in `context/sensitivity.py`
+- `BuildStats.dropped_reasons["sensitivity"]` tracks sensitivity-dropped item count
 - `.pre-commit-config.yaml` with ruff format, ruff check --fix, and standard file hygiene hooks
 
 ## [0.1.1] - 2026-03-03

CLAUDE.md

@@ -36,7 +36,7 @@ _utils.py       → text similarity (tokenize, jaccard, TfIdfScorer)
 serde.py        → to_dict/from_dict helpers
 store/          → in-memory data stores (append-only event log, artifact store, …)
 summarize/      → rule engine + fact extraction
-context/        → full context compilation pipeline
+context/        → full context compilation pipeline (incl. sensitivity.py for sensitivity enforcement)
 routing/        → catalog, DAG, beam-search router, card renderer
 adapters/       → MCP and A2A protocol conversion
 __main__.py     → CLI entry point

docs/architecture.md

@@ -41,22 +41,24 @@ the "context window problem" for tool-using AI agents.
 ## Context Engine pipeline
 
 The Context Engine compiles a phase-aware, budget-constrained prompt from
-the event log. The pipeline has seven stages:
+the event log. The pipeline has eight stages:
 
 1. **generate_candidates** — pull events from the event log and inject
    episodic memory and facts into the candidate pool.
 2. **dependency_closure** — if a selected item has a `parent_id`, bring
    the parent along even if it scored lower.
-3. **apply_firewall** — large tool results (above threshold) are
+3. **sensitivity_filter** — drop or redact items whose `sensitivity`
+   level meets or exceeds `ContextPolicy.sensitivity_floor`.
+4. **apply_firewall** — large tool results (above threshold) are
    summarised; the raw output is stored in the ArtifactStore and replaced
    with a compact reference + summary.
-4. **score_candidates** — rank candidates by recency, tag match, kind
+5. **score_candidates** — rank candidates by recency, tag match, kind
    priority, and token cost.
-5. **deduplicate_candidates** — remove near-duplicate items using Jaccard
+6. **deduplicate_candidates** — remove near-duplicate items using Jaccard
    similarity over tokenised text.
-6. **select_and_pack** — greedily pack the highest-scoring candidates
+7. **select_and_pack** — greedily pack the highest-scoring candidates
    into the token budget for the current phase.
-7. **render_context** — assemble the final prompt string, grouped by
+8. **render_context** — assemble the final prompt string, grouped by
    section (facts, history, tool results), with `BuildStats` metadata.
 
 ## Routing Engine pipeline

docs/concepts.md

@@ -65,6 +65,23 @@ A `ResultEnvelope` captures the processed output of a tool call:
 - `views` — optional alternative representations.
 - `status` — success / error / partial.
 
+## Sensitivity Enforcement
+
+Each `ContextItem` has a `sensitivity` field (default: `public`) that
+classifies its data sensitivity level. The `ContextPolicy.sensitivity_floor`
+setting (default: `confidential`) determines which items are subject to
+filtering during context compilation.
+
+Items whose sensitivity level meets or exceeds the floor are either:
+
+- **Dropped** (`sensitivity_action="drop"`, the default) — removed from
+  the candidate list before scoring or rendering.
+- **Redacted** (`sensitivity_action="redact"`) — text replaced with
+  `[REDACTED: {sensitivity}]` via the `MaskRedactionHook`, while
+  preserving all item metadata.
+
+Dropped items are recorded in `BuildStats.dropped_reasons["sensitivity"]`.
+
 ## Build Stats
 
 Every context build produces a `BuildStats` object that explains exactly

src/contextweaver/__init__.py

@@ -24,6 +24,7 @@
 from contextweaver._utils import TfIdfScorer, jaccard
 from contextweaver.config import ContextBudget, ContextPolicy, ScoringConfig
 from contextweaver.context.manager import ContextManager
+from contextweaver.context.sensitivity import MaskRedactionHook, register_redaction_hook
 from contextweaver.envelope import (
     BuildStats,
     ChoiceCard,
@@ -134,6 +135,8 @@
     "StoreBundle",
     # context engine
     "ContextManager",
+    "MaskRedactionHook",
+    "register_redaction_hook",
     # routing engine
     "Catalog",
     "ChoiceGraph",

src/contextweaver/config.py

@@ -101,7 +101,9 @@ class ContextPolicy:
         ttl_behavior: How to handle items that have exceeded their TTL.
             ``"drop"`` removes them; ``"warn"`` keeps them but fires a hook.
         sensitivity_floor: Items at or above this sensitivity level are
-            subject to redaction hooks before being included.
+            dropped or redacted (depending on ``sensitivity_action``).
+        sensitivity_action: ``"drop"`` (default) removes items at or above
+            the floor; ``"redact"`` replaces their text via redaction hooks.
         redaction_hooks: Names of redaction hook implementations to apply,
             in order.  Resolved at runtime by the context manager.
     """
@@ -116,5 +118,6 @@ class ContextPolicy:
     )
     ttl_behavior: str = "drop"
     sensitivity_floor: Sensitivity = Sensitivity.confidential
+    sensitivity_action: str = "drop"
     redaction_hooks: list[str] = field(default_factory=list)
     extra: dict[str, Any] = field(default_factory=dict)

src/contextweaver/context/__init__.py

@@ -15,11 +15,19 @@
 from contextweaver.context.prompt import render_context, render_item
 from contextweaver.context.scoring import score_candidates, score_item
 from contextweaver.context.selection import select_and_pack
+from contextweaver.context.sensitivity import (
+    MaskRedactionHook,
+    apply_sensitivity_filter,
+    register_redaction_hook,
+)
 
 __all__ = [
     "ContextManager",
+    "MaskRedactionHook",
     "apply_firewall",
     "apply_firewall_to_batch",
+    "apply_sensitivity_filter",
+    "register_redaction_hook",
     "build_schema_header",
     "deduplicate_candidates",
     "generate_candidates",

src/contextweaver/context/manager.py

@@ -4,11 +4,12 @@
 
 1. :func:`~contextweaver.context.candidates.generate_candidates` — phase filter
 2. :func:`~contextweaver.context.candidates.resolve_dependency_closure` — parent chain expansion
-3. :func:`~contextweaver.context.firewall.apply_firewall_to_batch` — raw output interception
-4. :func:`~contextweaver.context.scoring.score_candidates` — relevance scoring
-5. :func:`~contextweaver.context.dedup.deduplicate_candidates` — near-duplicate removal
-6. :func:`~contextweaver.context.selection.select_and_pack` — budget-aware selection
-7. :func:`~contextweaver.context.prompt.render_context` — prompt assembly
+3. :func:`~contextweaver.context.sensitivity.apply_sensitivity_filter` — sensitivity enforcement
+4. :func:`~contextweaver.context.firewall.apply_firewall_to_batch` — raw output interception
+5. :func:`~contextweaver.context.scoring.score_candidates` — relevance scoring
+6. :func:`~contextweaver.context.dedup.deduplicate_candidates` — near-duplicate removal
+7. :func:`~contextweaver.context.selection.select_and_pack` — budget-aware selection
+8. :func:`~contextweaver.context.prompt.render_context` — prompt assembly
 """
 
 from __future__ import annotations
@@ -22,6 +23,7 @@
 from contextweaver.context.prompt import render_context
 from contextweaver.context.scoring import score_candidates
 from contextweaver.context.selection import select_and_pack
+from contextweaver.context.sensitivity import apply_sensitivity_filter
 from contextweaver.envelope import ContextPack, ResultEnvelope
 from contextweaver.protocols import (
     ArtifactStore,
@@ -351,7 +353,7 @@ def _build(
     ) -> ContextPack:
         """Run the full context compilation pipeline (synchronous core).
 
-        All seven pipeline steps are pure computation, so no ``await`` is
+        All eight pipeline steps are pure computation, so no ``await`` is
         needed.  Both :meth:`build` (async) and :meth:`build_sync` delegate
         here.
 
@@ -387,15 +389,18 @@ def _build(
         # 2. Dependency closure
         candidates, closures = resolve_dependency_closure(candidates, self._event_log)
 
-        # 3. Firewall
+        # 3. Sensitivity filter
+        candidates, sensitivity_drops = apply_sensitivity_filter(candidates, self._policy)
+
+        # 4. Firewall
         candidates, envelopes = apply_firewall_to_batch(
             candidates, self._artifact_store, self._hook
         )
 
-        # 4. Score
+        # 5. Score
         scored = score_candidates(candidates, query, _tags, self._scoring)
 
-        # 5. Dedup
+        # 6. Dedup
         scored, dedup_removed = deduplicate_candidates(scored)
 
         # Pre-build episodic + fact injection text so we can estimate its
@@ -462,13 +467,22 @@ def _build(
         else:
             adjusted = effective_budget
 
-        # 6. Select (budget already accounts for header/footer overhead)
+        # 7. Select (budget already accounts for header/footer overhead)
         selected, stats = select_and_pack(scored, phase, adjusted, self._policy, self._estimator)
         stats.dedup_removed = dedup_removed
         stats.dependency_closures = closures
         stats.header_footer_tokens = hf_tokens
+        if sensitivity_drops > 0:
+            # Account for items dropped by sensitivity filtering in both the
+            # total candidate count and the drop breakdown so that
+            # dropped_count + included_count <= total_candidates remains true.
+            stats.total_candidates += sensitivity_drops
+            stats.dropped_count += sensitivity_drops
+            stats.dropped_reasons["sensitivity"] = (
+                stats.dropped_reasons.get("sensitivity", 0) + sensitivity_drops
+            )
 
-        # 7. Render
+        # 8. Render
         prompt = render_context(selected, header=full_header, footer=footer)
 
         pack = ContextPack(prompt=prompt, stats=stats, phase=phase, envelopes=envelopes)

src/contextweaver/context/sensitivity.py

@@ -0,0 +1,155 @@
+"""Sensitivity enforcement for the contextweaver Context Engine.
+
+Filters or redacts :class:`~contextweaver.types.ContextItem` objects whose
+sensitivity level meets or exceeds the configured
+:attr:`~contextweaver.config.ContextPolicy.sensitivity_floor`.
+
+Two actions are supported:
+
+* **drop** (default) — the item is silently removed from the candidate list.
+* **redact** — the item's text is replaced with a placeholder via a
+  :class:`~contextweaver.protocols.RedactionHook`.
+
+The built-in :class:`MaskRedactionHook` replaces the text with
+``[REDACTED: {sensitivity}]`` while preserving all other item metadata.
+"""
+
+from __future__ import annotations
+
+from dataclasses import replace
+
+from contextweaver.config import ContextPolicy
+from contextweaver.protocols import RedactionHook
+from contextweaver.types import ContextItem, Sensitivity
+
+# Ordered severity levels for comparison.
+_SENSITIVITY_ORDER: dict[Sensitivity, int] = {
+    Sensitivity.public: 0,
+    Sensitivity.internal: 1,
+    Sensitivity.confidential: 2,
+    Sensitivity.restricted: 3,
+}
+
+# Valid values for ContextPolicy.sensitivity_action.
+_VALID_ACTIONS: set[str] = {"drop", "redact"}
+
+# Hook registry (name → instance).  Built-in hooks are registered at
+# module load time; users can add their own via :func:`register_redaction_hook`.
+_HOOK_REGISTRY: dict[str, RedactionHook] = {}
+
+
+def register_redaction_hook(name: str, hook: RedactionHook) -> None:
+    """Register a custom :class:`~contextweaver.protocols.RedactionHook`.
+
+    Once registered, the *name* can be used in
+    :attr:`~contextweaver.config.ContextPolicy.redaction_hooks` just like the
+    built-in ``"mask"`` hook.
+
+    Args:
+        name: Short identifier for the hook (e.g. ``"my_custom_hook"``).
+        hook: An object implementing the :class:`RedactionHook` protocol.
+
+    Raises:
+        ValueError: If *name* is already registered.
+    """
+    if name in _HOOK_REGISTRY:
+        msg = f"Redaction hook {name!r} is already registered"
+        raise ValueError(msg)
+    _HOOK_REGISTRY[name] = hook
+
+
+class MaskRedactionHook:
+    """Replace item text with ``[REDACTED: {sensitivity}]``.
+
+    All other item fields (id, kind, metadata, parent_id, artifact_ref) are
+    preserved so the item still participates in dependency closure, stats
+    tracking, and rendering structure.
+    """
+
+    def redact(self, item: ContextItem) -> ContextItem:
+        """Return a copy of *item* with its text replaced by a redaction mask.
+
+        Args:
+            item: The context item to redact.
+
+        Returns:
+            A new :class:`ContextItem` with masked text and a minimal
+            token estimate.
+        """
+        placeholder = f"[REDACTED: {item.sensitivity.value}]"
+        return replace(item, text=placeholder, token_estimate=len(placeholder) // 4)
+
+
+# Register the built-in hook so it can be referenced by name in
+# ContextPolicy.redaction_hooks.
+_HOOK_REGISTRY["mask"] = MaskRedactionHook()
+
+
+def _resolve_hooks(names: list[str]) -> list[RedactionHook]:
+    """Resolve hook names to instances.
+
+    Args:
+        names: Hook names from :attr:`ContextPolicy.redaction_hooks`.
+
+    Returns:
+        Resolved :class:`RedactionHook` instances.
+
+    Raises:
+        ValueError: If a name cannot be resolved.
+    """
+    hooks: list[RedactionHook] = []
+    for name in names:
+        hook = _HOOK_REGISTRY.get(name)
+        if hook is None:
+            msg = f"Unknown redaction hook {name!r}. Available: {sorted(_HOOK_REGISTRY)}"
+            raise ValueError(msg)
+        hooks.append(hook)
+    return hooks
+
+
+def apply_sensitivity_filter(
+    items: list[ContextItem],
+    policy: ContextPolicy,
+) -> tuple[list[ContextItem], int]:
+    """Filter or redact items whose sensitivity meets or exceeds the policy floor.
+
+    Args:
+        items: Candidate items to inspect.
+        policy: The active context policy (provides ``sensitivity_floor``,
+            ``sensitivity_action``, and ``redaction_hooks``).
+
+    Returns:
+        A 2-tuple ``(filtered_items, dropped_count)``.  In ``"redact"`` mode
+        *dropped_count* is always ``0`` because items are kept (with masked
+        text).
+    """
+    floor_level = _SENSITIVITY_ORDER[policy.sensitivity_floor]
+    action = policy.sensitivity_action
+
+    if action not in _VALID_ACTIONS:
+        msg = f"Unknown sensitivity_action {action!r}. Valid: {sorted(_VALID_ACTIONS)}"
+        raise ValueError(msg)
+
+    # Resolve redaction hooks once (only needed in redact mode).
+    hooks: list[RedactionHook] = []
+    if action == "redact":
+        hook_names = policy.redaction_hooks or ["mask"]
+        hooks = _resolve_hooks(hook_names)
+
+    result: list[ContextItem] = []
+    dropped = 0
+    for item in items:
+        item_level = _SENSITIVITY_ORDER[item.sensitivity]
+        if item_level >= floor_level:
+            if action == "redact":
+                redacted = item
+                for hook in hooks:
+                    redacted = hook.redact(redacted)
+                result.append(redacted)
+            else:
+                # Default: drop
+                dropped += 1
+        else:
+            result.append(item)
+
+    return result, dropped