fix(vectordb): share single adapter across account backends to prevent RocksDB lock contention#777
Merged
zhoujh01 merged 1 commit intovolcengine:mainfrom Mar 19, 2026
Conversation
…t RocksDB lock contention
## Problem
When using the `local` vectordb backend in multi-tenant mode, every call to
`_get_backend_for_account()` or `_get_root_backend()` creates a new
`_SingleAccountBackend`, which in turn calls `create_collection_adapter(config)`.
For the local backend, each adapter creates its own `PersistStore` instance via
`get_or_create_local_collection()` → `create_store_manager("local", path)` →
`engine.PersistStore(path)`. Since all account backends use the **same storage
path**, multiple `PersistStore` instances attempt to acquire an exclusive RocksDB
lock on the same `LOCK` file at `{path}/store/LOCK`.
The first instance succeeds, but all subsequent instances fail with:
RuntimeError: IO error: lock /app/data/vectordb/context/store/LOCK:
already held by process
This makes **all vector operations permanently broken** after the second account
backend is created — search returns zero results, upserts silently fail to
index, and the error floods the logs on every request.
The bug is triggered in any deployment with more than one account or when both
a regular account backend and the root backend are accessed (which happens on
almost every API request).
## Root cause
`VikingVectorIndexBackend` lazily creates per-account `_SingleAccountBackend`
instances. Each one independently calls `create_collection_adapter(config)`,
producing separate `LocalCollectionAdapter` → `PersistStore` instances that all
point to the same on-disk RocksDB directory. RocksDB enforces single-process
exclusive locking, so only the first opener succeeds.
## Fix
Create the adapter **once** in `VikingVectorIndexBackend.__init__()` and pass it
as a `shared_adapter` to every `_SingleAccountBackend`. The adapter (and its
underlying `PersistStore` / RocksDB handle) is reused across all account
backends, eliminating lock contention entirely.
The `_SingleAccountBackend` constructor now accepts an optional `shared_adapter`
parameter. When provided, it uses the shared instance instead of creating a new
one. When not provided (e.g. standalone usage), it falls back to the original
`create_collection_adapter(config)` behavior for backward compatibility.
## Impact
- Fixes permanent vectordb failure in multi-tenant local backend deployments
- Search (`ov find`, `ov search`) now returns results
- Vector upserts during session commit are properly indexed
- Memory extraction vectorization completes successfully
- No behavioral change for single-account setups or non-local backends
ahmedhesham6
force-pushed
the
fix/vectordb-lock-contention
branch
from
81d2101 to
3362766
Compare
|
|
zhoujh01
approved these changes
Mar 19, 2026
zhoujh01
approved these changes
Mar 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fix RocksDB lock contention in the
localvectordb backend when running in multi-tenant mode. Every call to_get_backend_for_account()or_get_root_backend()was creating a new_SingleAccountBackendwith its ownPersistStoreinstance, all pointing to the same on-disk RocksDB directory. Since RocksDB enforces single-process exclusive locking, only the first opener succeeds — all subsequent ones fail withRuntimeError: IO error: lock ... already held by process.The fix creates the adapter once in
VikingVectorIndexBackend.__init__()and passes it as ashared_adapterto every_SingleAccountBackend, eliminating lock contention entirely.Related Issue
Type of Change
Changes Made
VikingVectorIndexBackend.__init__()instead of per-accountshared_adapterparameter to_SingleAccountBackendconstructor_SingleAccountBackendcreated by_get_backend_for_account()and_get_root_backend()shared_adapteris not provided, fall back tocreate_collection_adapter(config)Testing
Checklist
Screenshots (if applicable)
N/A
Additional Notes
VikingVectorIndexBackendmanages multiple account backends; standalone_SingleAccountBackendusage is unaffected