Skip to content

Implement Valkey/Redis exporter and tests for graph nodes & Appearance-Based ReID Tracking#5

Merged
bimantoromaesa merged 2 commits intomainfrom
feat_add_valkey_graph_node
Mar 9, 2026
Merged

Implement Valkey/Redis exporter and tests for graph nodes & Appearance-Based ReID Tracking#5
bimantoromaesa merged 2 commits intomainfrom
feat_add_valkey_graph_node

Conversation

@bimantoromaesa
Copy link
Contributor

What

v1.9.2 Beta — Valkey/Redis Graph Pipeline Storage + Appearance-Based ReID Tracking

Valkey / Redis Graph Pipeline Storage

  • Added export_valkey(result, url, key, ttl, serializer) — serializes any MATA result type to a Valkey or Redis key with optional TTL; supports json (default) and msgpack serializers
  • Added load_valkey(url, key, result_type="auto") — deserializes stored results back to their original type; auto-detects VisionResult, ClassifyResult, DepthResult, and OCRResult
  • Added publish_valkey(result, url, channel, serializer) — fire-and-forget Pub/Sub broadcast
  • Added _parse_valkey_uri() helper supporting valkey://, valkey://<db>/, and redis://user:pass@host:port/db/key URI formats
  • Added ValkeyStore graph sink node — pass-through sink with {node} / {timestamp} key template placeholders, TTL, and serializer selection
  • Added ValkeyLoad graph source node — inputs={} source that loads a stored result from Valkey and injects it as a typed artifact
  • Added valkey:// and redis:// URI scheme dispatch to all six result.save() methods (VisionResult, DetectResult, SegmentResult, ClassifyResult, DepthResult, OCRResult) — existing file-based paths fully unaffected
  • Added ModelRegistry.get_valkey_connection(name="default") — reads named connection profiles from storage.valkey YAML section; resolves password_env from environment variables
  • Added optional dependency groups: mata[valkey]valkey>=6.0.0; mata[redis]redis>=5.0.0
  • Exported export_valkey, load_valkey, publish_valkey from mata.core.exporters
  • Exported ValkeyStore, ValkeyLoad from mata.nodes
  • 89 new tests: test_valkey_exporter.py (42), test_valkey_nodes.py (33), test_valkey_config.py (14)

Appearance-Based ReID Tracking

  • Added mata.track(..., reid_model="org/model") — activates BotSort appearance re-identification via any HuggingFace encoder ID or local .onnx path
  • Added ReIDAdapter abstract base class — L2-normalised embedding output; lazy-loaded (zero startup overhead when unused)
  • Added HuggingFaceReIDAdapter — auto-detects CLIP / ViT-family / generic AutoModel architectures; all transformers imports lazy
  • Added ONNXReIDAdapter — auto-detects NCHW/NHWC input layout from model metadata; supports CPU and CUDA execution providers
  • Extended TrackingAdapter.update() to extract crops, batch-encode through the ReID encoder, and inject embeddings into BOTSORT; Instance.embedding populated in output VisionResult
  • Extended mata.track() with reid_model: str | None, with_reid: bool = False, and reid_bridge kwargs (backward-compatible defaults)
  • Added ReIDBridge — cross-camera appearance store backed by Valkey/Redis; publishes L2-normalised embeddings keyed by reid:{camera_id}:{track_id}; query() returns nearest cosine-similarity matches from other cameras; TTL-based auto-eviction; msgpack binary serialisation; scan_iter (non-blocking)
  • Extended TrackingAdapter.__init__() with reid_encoder and reid_bridge kwargs; confirmed-track embeddings published automatically after each update(); ConnectionError caught and logged, never raised
  • Exported ReIDAdapter, HuggingFaceReIDAdapter, ONNXReIDAdapter from mata.adapters
  • Exported ReIDBridge from mata.trackers
  • 80+ new tests: test_reid_adapter.py, test_tracking_reid.py, test_reid_bridge.py
  • Added examples/track/reid_tracking.py and examples/track/cross_camera_reid.py

Documentation

  • docs/VALKEY_GUIDE.md — full integration guide: installation, basic usage, graph pipeline, YAML config, streaming, Pub/Sub, security (TLS, password_env, SSRF prevention, key sanitization), performance tuning, and troubleshooting
  • docs/GRAPH_API_REFERENCE.md — new "Storage Nodes" section with parameter tables for ValkeyStore and ValkeyLoad
  • docs/VALIDATION_GUIDE.md — ReID tracking validation notes added
  • README.md — Valkey added to Key Features and Optional Dependencies; ReID tracking section with single-camera and cross-camera examples
  • QUICKSTART.md — new "Valkey / Redis Result Storage" section
  • QUICK_REFERENCE.md — new "Valkey/Redis Storage Quick Reference (v1.9)" cheatsheet

Changed

  • mata.nodes.__all__ extended with ValkeyStore and ValkeyLoad
  • mata.core.exporters.__init__ extended with export_valkey, load_valkey, publish_valkey
  • mata.track() signature extended with reid_model, with_reid, reid_bridge kwargs
  • TrackingAdapter.__init__() extended with reid_encoder, reid_bridge kwargs (both default None; zero overhead when unused)
  • BOTSORT.get_dists() appearance-distance branch now reachable when reid_encoder is set
  • src/mata/trackers/configs/botsort.yamlreid_model / with_reid documentation block added

Why

Closes two major feature tracks planned for v1.9.x:

  1. Valkey/Redis storage — enables MATA graph pipelines to persist and share results between nodes, services, and processes without writing intermediate files; supports both single-process caching and cross-service Pub/Sub streaming patterns.

  2. Appearance-Based ReID — extends BotSort tracking with embedding-based identity matching, dramatically reducing ID switches in occlusion-heavy scenes; ReIDBridge enables cross-camera identity correlation backed by the same Valkey infrastructure added in this release.


Checklist

  • Tests pass (pytest tests/ -v)
  • Linters pass (black --check src/ tests/ && ruff check src/)
  • Type checks pass (mypy src/mata/)
  • Documentation updated (if user-facing changes)
  • CHANGELOG.md updated (if notable change)

…ID Tracking.

- Implement comprehensive tests for the Valkey/Redis exporter, covering export and load functionalities for various result types including VisionResult, ClassifyResult, DepthResult, and OCRResult.
- Introduce tests for ValkeyStore and ValkeyLoad graph nodes, ensuring correct artifact handling, key templating, and integration within graph execution.
- Validate serialization and deserialization processes, including support for custom serializers and TTL parameters.
- Ensure robust error handling for missing keys and unsupported result types.
@bimantoromaesa bimantoromaesa merged commit 7603c12 into main Mar 9, 2026
4 checks passed
@bimantoromaesa bimantoromaesa deleted the feat_add_valkey_graph_node branch March 9, 2026 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant