v1.2.0

v1.2.0

Release Notes: 1.2.0

This document captures the net change from 1.1.1 to v1.2.0 as of March 27, 2026.
It reflects the current committed master branch after the 1.2.0 version bump and
excludes untracked local workspace files.

Current diff scope:

• 2 commits ahead of 1.1.1
• 3 files changed
• 19 insertions and 13 deletions

Executive Summary

v1.2.0 is a small API-focused release on top of 1.1.1. The net change set exposes
hybrid-search Reciprocal Rank Fusion tuning controls through the search API, aligns
the default ef request behavior with the server setting, and bumps the reported
server version string to 1.2.0.

This release is intended to work with Python Endee client version 0.1.23 or later.

The highest-impact changes for integrators are:

• /api/v1/search now accepts dense_rrf_weight
• /api/v1/search now accepts rrf_rank_constant
• invalid RRF tuning values now fail request validation with HTTP 400

Highlights

1. Hybrid-search RRF tuning is now part of the search API

• main() now accepts optional dense_rrf_weight and rrf_rank_constant in search
  requests.
• dense_rrf_weight must be between 0.0 and 1.0.
• rrf_rank_constant must be greater than 0.0.
• The dense and sparse blend is now derived at search time, with sparse weight
  computed as 1.0 - dense_rrf_weight.
• searchKNN() now receives these values directly instead of relying on local
  hard-coded constants.

Why this matters:

• Clients can tune hybrid ranking behavior per request instead of being locked to the
  previous fixed 0.5 / 0.5 weighting and rank constant 60.
• Integrators now have a stable API surface for experimentation and relevance tuning.
• Invalid tuning values fail early at the API boundary instead of silently falling
  through to search execution.

2. Search defaults are more explicit and centrally aligned

• The request handler now defaults ef to settings::DEFAULT_EF_SEARCH instead of
  passing 0.
• searchKNN() now also uses settings-backed defaults for ef,
  dense_rrf_weight, and rrf_rank_constant.
• settings.hpp now defines DEFAULT_DENSE_RRF_WEIGHT and
  DEFAULT_RRF_RANK_CONSTANT.

Why this matters:

• Search-request defaults now come from one configuration surface instead of being
  split between request parsing and local function constants.
• Future tuning changes are easier to make and audit because the defaults live in
  settings.

3. Versioning and client compatibility

• The backend version string is now 1.2.0.
• The intended minimum compatible Python Endee client version is 0.1.23.

Why this matters:

• Release packaging and runtime version reporting now align with the new release.
• Upgrade guidance is explicit for teams coordinating backend and Python client
  rollouts.

Operator-Facing Changes

Area	What changed in v1.2.0	Why teams should care
Search API	/api/v1/search now accepts dense_rrf_weight and rrf_rank_constant.	Clients can tune hybrid ranking without rebuilding the server.
Request validation	Invalid dense_rrf_weight and invalid rrf_rank_constant now return HTTP 400.	API consumers should validate payloads and update tests for the new contract.
Search defaults	ef now defaults directly to settings::DEFAULT_EF_SEARCH, and RRF defaults are settings-backed.	Request behavior is more explicit and easier to reason about across code paths.
Reported version	The backend version string is now 1.2.0.	Deployment metadata and server-reported version now match the release.
Python client compatibility	v1.2.0 is intended for Python Endee client 0.1.23 and later.	Upgrade plans should treat 0.1.23 as the minimum supported Python client baseline.

Upgrade Notes

1. Client request compatibility

• Existing search clients continue to work without sending the new fields.
• If you choose to send dense_rrf_weight, keep it within 0.0 to 1.0.
• If you choose to send rrf_rank_constant, keep it greater than 0.0.
• Update API tests and client-side validation to cover the new request parameters and
  their validation failures.

2. Staging checks worth doing

• Re-test hybrid search quality using your real dense and sparse workloads.
• Verify that clients depending on fixed hybrid weighting are still producing the
  expected ranking if they continue to use defaults.
• Validate application flows against Python Endee client 0.1.23 or newer before
  broad rollout.

Suggested Release Positioning

v1.2.0 is a focused search-tuning release for teams using hybrid retrieval. It
adds API-level control over RRF weighting and rank constants, makes search defaults
more explicit and settings-backed, sets the backend version to 1.2.0, and targets
Python Endee client 0.1.23 or later.

v1.1.1

v1.1.1

Release Notes: 1.1.1

This document captures the net change from 1.1.0 to v1.1.1 as of March 27, 2026.
It reflects the current committed master branch after the 1.1.1 version bump and
excludes untracked local workspace files.

Current diff scope:

• 11 commits ahead of 1.1.0
• 14 files changed
• 786 insertions and 338 deletions

Executive Summary

v1.1.1 is a stability and operability release built on top of 1.1.0. The net
change set does not introduce a broad new feature surface. Instead, it strengthens
index lifecycle safety, adds stricter startup validation around live-index and MDBX
map-size settings, changes the backup-list API response shape, improves sparse ingest
warnings, updates the packaged Web UI bundle to v1.4.0, and bumps the server
version string to 1.1.1.

This release is intended to work with the Python Endee client at version 0.1.22.

The highest-impact changes for operators and integrators are:

• live-index handling now relies on explicit dirty-state tracking and save-before-evict
  behavior
• startup now rejects invalid NDD_MAX_LIVE_INDICES and invalid MDBX map-size
  combinations
• /api/v1/backups no longer behaves like a flat filename listing

One commit in this range temporarily added rebuild-index support and a later commit
reverted it. v1.1.1 does not ship rebuild functionality, and the notes below refer
only to the final net state.

Highlights

1. Safer index lifecycle, eviction, and persistence

• CacheEntry now tracks is_dirty and last_dirtied_at instead of the older
  generic updated flag.
• Autosave, shutdown, reload, and eviction flows now work from explicit dirty state
  and save indices before removing them from the live cache.
• ensureLiveIndexCapacity() now enforces live-index capacity directly and raises an
  error if the cache remains full after eviction attempts.
• evictIfNeeded() now drops stale candidates from indices_list_, forces a save
  for dirty indices, and aborts eviction if an index remains dirty after that save.

Why this matters:

• In-memory mutations are better protected during eviction pressure, reloads, and
  controlled shutdowns.
• Multi-index deployments now get a clearer failure mode when live-cache capacity is
  exhausted.
• Operators should reason about residency and persistence together, rather than
  assuming the older memory-threshold behavior is still the effective control path.

2. Startup validation is stricter and the memory model is better documented

• main() now calls settings::validateSettings() before full server startup.
• NDD_MAX_LIVE_INDICES is validated against the documented ceiling.
• Each NDD_*_MAP_SIZE_BITS value must be less than its matching
  NDD_*_MAP_SIZE_MAX_BITS.
• Each NDD_*_MAP_SIZE_MAX_BITS value must stay within the compiled default ceiling.
• The settings helper now exposes the MDBX map-size controls alongside the rest of
  the server configuration surface.
• docs/memory_management.md documents the live-index model, MDBX virtual address
  space costs, and the PTHREAD_KEYS_MAX constraint behind the current live-index
  cap.

Why this matters:

• Invalid or oversized map-size overrides now fail fast instead of surviving until a
  harder-to-debug runtime issue.
• Live-index residency is now documented as an operational constraint, not just an
  implementation detail.
• This release improves safety and visibility even though it does not yet implement a
  true DRAM-aware eviction policy.

3. /api/v1/backups is now a metadata-backed JSON response

• BackupStore::listBackups() no longer scans the backup directory for .tar files
  and returns a plain string vector.
• The backup list path now reads the backup manifest JSON and returns that JSON
  directly.
• /api/v1/backups therefore returns a manifest-backed JSON payload instead of a
  filesystem-derived flat list of backup names.

Why this matters:

• This is a breaking response-shape change for scripts, SDKs, dashboards, and any
  automation that assumed a simple name list.
• Backup tooling can now consume metadata-backed backup records rather than deriving
  state from filenames alone.
• Integrators should treat backup listing as a compatibility checkpoint during
  rollout, not as a safe drop-in change.

4. Sparse ingest warnings and packaged frontend updates

• Sparse block updates now log zero and negative weights before the merge walk over
  new inserts.
• Zero weights are still treated as deleted entries and negative weights are still
  treated as dead entries; the change is about earlier diagnostics rather than a new
  sparse behavior model.
• install.sh now pulls Web UI bundle v1.4.0, up from the earlier v1.2.0
  baseline in 1.1.0.
• The branch also includes the backend version bump to 1.1.1.

Why this matters:

• Teams generating sparse payloads now get clearer warning signals when invalid
  weights enter the ingest path.
• Packaged installs receive a newer UI bundle, but the backend repo does not provide
  enough source evidence to claim specific frontend feature changes here.

Operator-Facing Changes

Area	What changed in v1.1.1	Why teams should care
Live-index capacity	Live-index admission and eviction are now governed by NDD_MAX_LIVE_INDICES and dirty-state persistence.	Operators should review cache-cap assumptions and validate behavior under multi-index pressure.
Legacy cache knobs	NDD_MAX_ACTIVE_INDICES and NDD_MAX_MEMORY_GB no longer define the current live-cache behavior.	Existing deployment configs may still contain knobs that no longer have the intended effect.
Startup validation	Invalid NDD_MAX_LIVE_INDICES and invalid or oversized MDBX map-size overrides now fail startup.	Environment mistakes become immediate boot failures instead of latent runtime surprises.
Backup list API	/api/v1/backups now returns manifest-backed JSON instead of a flat list of backup names discovered from .tar files.	Clients parsing the old shape may break until they are updated.
Index persistence	Dirty indices are saved before eviction, before reload, and during shutdown.	In-memory mutations are better protected during cache churn and controlled restarts.
Sparse ingest diagnostics	Zero and negative sparse weights are surfaced earlier in the new-insert path.	Invalid sparse inputs should now be easier to spot in logs.
Packaged frontend	install.sh now pulls Web UI v1.4.0.	Packaged deployments receive a newer frontend bundle.
Reported version	The backend version string is now 1.1.1.	Release packaging and version reporting now match the release notes.
Python client compatibility	v1.1.1 works with Python Endee client 0.1.22.	Client and server version guidance is explicit for teams planning upgrades.

Upgrade Notes

1. Backup-list compatibility

• Update any client that expects /api/v1/backups to return a plain array of backup
  names.
• Treat this as a contract change rather than an internal refactor.
• Re-test backup listing, backup dashboards, and any restore workflow that reads from
  the list response before promoting this release broadly.

2. Server configuration compatibility

• Move any operational reliance from NDD_MAX_ACTIVE_INDICES to
  NDD_MAX_LIVE_INDICES.
• Do not rely on NDD_MAX_MEMORY_GB to control the current eviction path.
• Validate all MDBX map-size overrides before rollout:
  NDD_INDEX_META_MAP_SIZE_BITS,
  NDD_ID_MAPPER_MAP_SIZE_BITS,
  NDD_FILTER_MAP_SIZE_BITS,
  NDD_METADATA_MAP_SIZE_BITS,
  NDD_VECTOR_MAP_SIZE_BITS, and their corresponding *_MAX_BITS values.
• Expect startup to abort if any *_BITS >= *_MAX_BITS, if any max override exceeds
  the compiled ceiling, or if NDD_MAX_LIVE_INDICES exceeds the documented cap.

3. Staging checks worth doing

• Exercise multi-index workloads that create or load enough indices to hit
  NDD_MAX_LIVE_INDICES.
• Confirm that dirty indices are saved correctly during eviction, explicit reloads,
  and controlled shutdowns.
• Review logs for sparse zero and negative weight warnings if your ingest path emits
  sparse payloads automatically.
• Verify all backup-list consumers against the manifest-backed JSON response before
  upgrading automation.
• Validate application flows against Python Endee client 0.1.22 if that client is
  part of your deployment path.

Suggested Release Positioning

v1.1.1 is a stability and operability follow-up to 1.1.0, centered on safer
live-index eviction and persistence, stricter startup validation for live-index and
MDBX settings, a breaking but more metadata-rich backup-list API, clearer sparse
ingest diagnostics, a packaged Web UI update to v1.4.0, compatibility with Python
Endee client 0.1.22, and an aligned backend version bump to 1.1.1.

v1.1.0

Release Notes: master vs 1.0.0

This note describes the current master branch relative to the 1.0.0 release baseline.

Current diff scope at the time of writing:

• 3 commits ahead of 1.0.0
• 11 files changed
• 724 insertions and 436 deletions

Executive Summary

master is a minor follow-up to 1.0.0. The practical center of the diff is the sparse and hybrid retrieval contract: sparse indexes are now configured by sparse_model instead of sparse_dim, sparse search adds a new endee_bm25 mode with server-side IDF weighting, and sparse upserts now behave as full replacements so stale postings and sparse document counts stay aligned.

Around that core change, the branch updates index metadata, backup metadata, and index-info APIs to report sparse_model, rewrites the getting-started guide around Docker-first onboarding, and bumps the server version string to 1.1.0. Compatible with python client version 0.1.21 onwards.

Highlights

1. Sparse configuration moved from dimensions to scoring models

• Sparse-capable indexes now carry an explicit SparseScoringModel enum with None, default, and endee_bm25.
• Create-index now rejects legacy sparse fields such as sparse_dim and sparse_scoring_model.
• Dense-only indexes can still omit sparse configuration; the server defaults them to None.
• Index list, index info, backup metadata, and persisted metadata now expose sparse_model instead of sparse_dim.

Why it matters:

• The API now describes sparse behavior in ranking terms instead of storage-shape terms.
• Clients can distinguish dense-only, default sparse, and BM25-style sparse search explicitly.
• Older automation that still sends or reads sparse_dim will need updating.

2. Sparse upserts and deletes are safer

• Sparse writes now treat hybrid ingests as document replacements instead of append-like updates.
• When a document is updated, old sparse postings are removed before new postings are written.
• Empty sparse payloads now clear existing sparse state instead of leaving stale terms behind.
• The in-memory sparse document count is updated only after transaction commit and is rebuilt on startup by counting only non-empty sparse rows.

Why it matters:

• Re-ingesting the same ID no longer risks stale sparse postings persisting in the inverted index.
• Sparse search statistics now track live sparse-bearing documents more accurately.
• The new counting path is a prerequisite for the server-side IDF logic introduced in this branch.

3. Hybrid ranking now has explicit scoring-model plumbing

• Sparse search now accepts the live sparse document count and computes a BM25-style IDF factor for endee_bm25.
• Hybrid fusion has been restructured around weighted reciprocal rank fusion (RRF).
• The current weights are hard-coded equally for dense and sparse lists, but the fusion path now supports per-source weighting cleanly.
• Sparse queries reuse caller ordering when already sorted and continue to share filter bitmaps with the dense path.

Why it matters:

• endee_bm25 can change sparse and hybrid ranking materially versus the earlier default sparse path.
• The fusion logic is now structured for future tuning without another search-path rewrite.
• Query-time work is cleaner and better aligned across dense and sparse branches.

4. Onboarding docs were rewritten for simpler adoption

• docs/getting-started.md was rewritten as a user-first guide instead of a build-only reference.
• The guide now centers four entry points: Docker, Docker build from source, install.sh, and manual CMake.
• Windows guidance is now explicit: Docker is the supported path.
• README.md now points release-note links at the GitHub 1.0.0 release page.

Why it matters:

• New users get a clearer first-run path.
• Docker-first guidance is now much more visible.
• The docs are better aligned with how most teams will evaluate the project.

Operator-Facing Changes

Area	What changed now	Why teams should care
Create-index API	Sparse configuration uses sparse_model; legacy sparse_dim and sparse_scoring_model inputs are rejected	Existing client payloads may start failing with 400 until updated
Index metadata	Persisted metadata requires sparse_model	Older 1.0.0 indexes do not load cleanly on current master
Backup metadata	Backup creation writes sparse_model, and restore expects it	Older backup archives should be validated before treating restore as seamless
Index list/info APIs	/api/v1/index/list and /api/v1/index/<name>/info now return sparse_model	SDKs, dashboards, and scripts parsing sparse_dim need adjustment
Sparse ingest semantics	Sparse upserts now replace prior sparse state and empty sparse vectors clear old postings	Hybrid relevance should be more stable under repeated writes
Sparse scoring	New endee_bm25 mode multiplies query term weights by server-side IDF	Relevance should be revalidated if you adopt the new sparse model
Versioning	settings::VERSION is now 1.1.0	Packaging and release automation should treat this branch as the next minor line

Upgrade Notes

Sparse metadata compatibility

The current metadata loader fails fast if sparse_model is missing or invalid. In practice, that means 1.0.0 index metadata is not treated as forward-compatible.

Practical guidance:

• Plan to rebuild or re-ingest indexes created under 1.0.0.
• Treat this as a release-blocking migration item, not a minor schema rename.
• Validate backup restore behavior against real 1.0.0 archives before calling the upgrade seamless.

API contract changes

• Update create-index clients to send sparse_model when sparse search is needed.
• Update index-management tooling to read sparse_model from list/info payloads.
• Remove any reliance on sparse_dim or sparse_scoring_model in request builders and response parsers.

Relevance checks worth doing in staging

• Compare default versus endee_bm25 sparse ranking on representative workloads.
• Re-test repeated hybrid upserts where sparse terms are removed or replaced.
• Re-test delete flows for hybrid documents to confirm sparse postings disappear as expected.

Suggested Release Positioning

A sparse and hybrid retrieval contract update over 1.0.0, with the headline changes being sparse_model, safer sparse replacement semantics, and BM25-style sparse scoring support, plus a much more approachable onboarding guide.

1.0.0

Release Notes: 1.0.0

This note describes the 1.0.0 release relative to the 1.0.0-beta baseline.

Current diff scope at the time of writing:

• 78 commits ahead of 1.0.0-beta
• 47 files changed
• 8,710 insertions and 3,686 deletions

Executive Summary

1.0.0 is not a small follow-up to 1.0.0-beta. The current branch is a substantial platform update focused on sparse retrieval, filtered search, backup operations, observability, and build/runtime ergonomics. Compatible with python client version 0.1.19 or before.

The largest practical change is the sparse subsystem rewrite: the earlier implementation has been replaced with a dedicated MDBX-backed inverted index with explicit on-disk version checks, configurable search batching, and better compaction behavior. Around that, the code now adds safer backup workflows, more tunable filtering, stronger operational logging, and a cleaner build/test story.

Highlights

1. Sparse search has been rebuilt

• The sparse engine moved from the older bmw.hpp path to a dedicated inverted_index implementation.
• Sparse postings are now stored as MDBX-backed blocked posting lists with per-term metadata and a superblock for format validation.
• Search batching is runtime configurable through NDD_INV_IDX_SEARCH_BATCH_SZ.
• The sparse path now supports optional float storage via NDD_INV_IDX_STORE_FLOATS, while the default path keeps compact quantized storage.
• The current implementation also documents and enforces sparse format compatibility instead of silently opening legacy data.

Why it matters:

• Better control over sparse search cost.
• Clearer recovery behavior for incompatible on-disk layouts.
• A storage layout that is easier to reason about and instrument.

2. Filtered and hybrid search are more capable

• Filtering is now documented and structured around numeric, category, and boolean paths in filter.md.
• Search requests now accept filter_params, including prefilter_threshold and boost_percentage, so filtered search can be tuned per request.
• JSON vector inserts now support meta, filter, and norm in addition to dense and sparse vector payloads.
• Filter tests were added and can be built with -DENABLE_TESTING=ON.

Why it matters:

• More predictable filtered retrieval behavior.
• Better support for hybrid dense+sparse workloads.
• A clearer path for validating filter regressions during release testing.

3. Backups are now async, user-scoped, and safer operationally

• Backup creation now runs asynchronously and returns 202 Accepted instead of blocking the request until archive creation finishes.
• Backup storage moved to per-user directories under {DATA_DIR}/backups/{username}/.
• Archive creation now stages in a temp directory and renames into place atomically.
• New backup endpoints were added for download, upload, active-status checks, and metadata inspection.
• Backup upload support exists, but the current multipart path still buffers the uploaded archive in memory before writing it to disk.
• Backup implementation details are documented in backup-system.md.

New or expanded backup API surface:

• POST /api/v1/index/{name}/backup
• GET /api/v1/backups
• GET /api/v1/backups/active
• GET /api/v1/backups/{name}/info
• GET /api/v1/backups/{name}/download
• POST /api/v1/backups/upload
• POST /api/v1/backups/{name}/restore
• DELETE /api/v1/backups/{name}

Why it matters:

• Long-running backup work no longer ties up the request thread.
• Backup lifecycle is visible to clients.
• Temp-file cleanup and atomic rename behavior reduce partial-backup risk.

4. Runtime behavior and observability improved

• Logs were standardized around LOG_INFO, LOG_WARN, and LOG_ERROR, with stable numeric codes and explicit username/index_name context in logs.md.
• MDBX timing instrumentation was added behind ND_MDBX_INSTRUMENT; sparse timing instrumentation is also available behind ND_SPARSE_INSTRUMENT.
• Server thread count is now runtime configurable through NDD_NUM_SERVER_THREADS.
• Vector cache controls were added through NDD_VECTOR_CACHE_PERCENTAGE and NDD_VECTOR_CACHE_MIN_BITS.
• The HNSW path picked up graph-backfill and cache-related work intended to improve search/load behavior on active indexes.

Why it matters:

• Operators get much better failure context in production logs.
• Performance investigations can be done without carrying permanent instrumentation overhead.
• More runtime knobs are available without recompiling.

5. Build and developer workflow are cleaner

• Sparse sources are compiled into a separate object library, which improves parallel compilation behavior.
• ENABLE_TESTING=ON now builds a dedicated ndd_filter_test target.
• run.sh was added to auto-detect the built binary and run with NDD_DATA_DIR and optional auth token.
• install.sh was expanded with clearer OS/package handling and frontend asset setup.
• README and subsystem docs were significantly expanded.

Operator-Facing Changes

Area	What changed now	Why teams should care
Sparse storage	Sparse indexes now use a superblock-validated on-disk format	Older sparse data may not open without rebuild/reingest
Backups	Archive format is now .tar in per-user backup directories	Existing tooling that expected global .tar.gz archives needs updating
Create-index defaults	Default precision now resolves to int16 unless explicitly provided	Memory/latency characteristics may differ from beta defaults
Search tuning	filter_params.prefilter_threshold and boost_percentage are supported	Filter-heavy queries can be tuned without code changes
Threading	Server concurrency is runtime-configurable with NDD_NUM_SERVER_THREADS	Default runtime behavior may differ from earlier deployments
Cache tuning	Vector cache sizing can be adjusted by env var	Memory usage can be tuned per deployment
Logs	Log format is now standardized and code-based	Easier alerting, parsing, and support workflows

Upgrade Notes

Sparse index compatibility

The current sparse implementation explicitly checks for a sparse superblock and format version. If an older sparse database exists without that metadata, the current code will reject it as incompatible rather than attempting a best-effort open.

Practical guidance:

• Plan to rebuild or re-ingest sparse indexes created before the new superblock format.
• Treat sparse on-disk compatibility as a release note item, not an implementation detail.

Backup format and layout changes

1.0.0-beta used global .tar.gz backup archives. The current code expects per-user .tar archives and adds temp staging plus active-job tracking.

Practical guidance:

• Review any scripts that assume .tar.gz.
• Review any scripts that assume a single global backup directory.
• Validate restore behavior against real beta-era archives before calling the upgrade seamless.

API behavior changes

• Backup creation is now asynchronous and returns a progress state instead of a finished archive immediately.
• Backup restore now returns 201 Created on success.
• Backup delete now returns 204.
• Download uses the backup file directly from disk and currently expects .tar.
• Search can take request-level filter tuning through filter_params.
• JSON inserts accept meta, filter, and norm.

Default and tuning changes worth checking in staging

• Default create-index precision changed from the earlier INT8 default path to int16.
• int8d and int16d legacy names are normalized to int8 and int16.
• Default filter prefilter threshold moved from 1000 to 10000.
• Default server threading now comes from runtime settings; when NDD_NUM_SERVER_THREADS is unset, the code resolves it to roughly 2 x hardware_concurrency().

Suggested Release Positioning

This release should be positioned as:

A storage and operations upgrade over 1.0.0-beta, with the headline changes being the new sparse engine, safer backup lifecycle, better filtered retrieval controls, and stronger production observability.

Related Docs

• Sparse vector search
• Backup system
• Filter design
• Log format
• MDBX instrumentation

1.0.0-beta

License (#4)