feat(storage): add path locking and selective crash recovery for write operations

[qin-ctx]

Description

Implement path-level locking and selective crash recovery for VikingFS storage operations. Core write operations (rm, mv, add_resource, session.commit) are coordinated through LockManager with fencing-token-based path locks and a RedoLog for session memory crash recovery.

Note

This PR went through a major design iteration: the initial undo-based TransactionManager/Journal/UndoLog system (~4000 lines) was replaced with a lightweight LockManager + RedoLog architecture (~700 lines), reducing complexity by ~82%. See commit 4e44a5d for the refactor rationale.

cc @r266-tech

Related Issue

Closes #390

RFC Discussion: #115

Type of Change

• Bug fix
• New feature (non-breaking change that adds functionality)
• Documentation update
• Test update

Scope & Limitations

Important

This design is for single-node deployments only and does not support multi-node/distributed scenarios.

Component	Current Approach	Multi-node Requirements
PathLock	File-based locks (fencing token in .path.ovlock)	Distributed locks (etcd/ZooKeeper) with atomic CAS + session-bound leases
RedoLog	AGFS-persisted markers at /local/_system/redo/	Persist to distributed consistent store
QueueFS (SQLite)	SQLite + WAL mode, RecoverStale on startup	Distributed message queue or TiDB/MySQL backend

Architecture

Overview

LockContext (async context manager — user-facing entry point)
  └── LockManager (global singleton)
        ├── PathLock (fencing-token-based path locks via .path.ovlock files)
        ├── RedoLog (AGFS-persisted crash recovery for session_memory)
        └── Background cleanup (stale lock detection every 60s)

Design Decisions

1. Inline error handling, not rollback: VikingFS rm/mv use sequential operations with explicit error cleanup instead of undo-log rollback. Failures are observable, not hidden.

2. Selective crash recovery (RedoLog): Only session.commit memory extraction is redo-protected — the one operation where data loss is unacceptable and the operation is idempotent enough to safely replay. General FS operations don't need redo because partial failures are detectable.

3. Fencing tokens for locks: Lock files contain {owner_id}:{monotonic_ns}:{lock_type} for stale lock detection and ABA prevention.

4. Fail-fast by default: lock_timeout=0 means operations fail immediately on lock conflict. Set to positive value for blocking retries.

Lock Design

Two lock types:

Lock Type	Purpose	Conflict Detection
POINT (P)	Single directory (write/semantic)	Checks ancestors for SUBTREE locks
SUBTREE (S)	Entire subtree (rm/mv-source)	Scans descendants for any locks

Three locking modes (LockContext):

Mode	Lock Combination	Used By
point	POINT on each path	add_resource, session.commit
subtree	SUBTREE on each path	rm (directories)
mv	SUBTREE on source + SUBTREE on destination	mv

Livelock prevention: Deterministic backoff via fencing token timestamp comparison.

Stale lock cleanup: Background loop (every 60s) detects locks held >3600s; lock_expire config (default 300s) enables force-release on acquisition conflict.

Per-Command Design

Command	Lock Mode	Error Handling
rm()	Dir: subtree; File: point(parent)	VectorDB delete failure → log warning, still delete FS (eventual consistency)
mv()	mv: SUBTREE(src) + SUBTREE(dst)	VectorDB update failure → delete copy, propagate error (source untouched)
add_resource (finalize)	point(parent dir)	FS or enqueue failure → propagate error
session.commit	Redo-log protected (no lock on archive phase)	Phase 1 failure → no redo marker; Phase 2 failure → redo marker persists for recovery

Session Commit — Two-Phase with Redo Protection

Phase 1 (Archive — no lock):
  increment index → generate summary → write archive → clear messages

Phase 2 (Memory — redo-protected):
  write redo marker → extract memories → write memories → enqueue semantic → mark done

If the process crashes during Phase 2, the redo marker persists. On restart, LockManager.start() scans /local/_system/redo/ and replays memory extraction.

Crash Recovery

On LockManager.start():

1. Scan /local/_system/redo/ for pending markers
2. For each pending task: re-extract memories from archived messages, write, enqueue semantic
3. Delete marker on success
4. Start background stale lock cleanup loop

Changes Made

• LockManager (lock_manager.py): Global singleton managing lock lifecycle, redo recovery, background stale cleanup
• LockContext (lock_context.py): Async context manager for lock acquisition/release
• LockHandle (lock_handle.py): Lock ownership tracking with LockOwner protocol
• PathLock (path_lock.py): Fencing-token-based path locks with POINT/SUBTREE types; fixed TOCTOU race in lock acquisition
• RedoLog (redo_log.py): Selective crash recovery for session_memory operations
• LockObserver (lock_observer.py): Monitoring for active/hanging/stale locks
• VikingFS: rm()/mv() use LockContext with inline error handling
• ResourceProcessor: add_resource finalize phase uses point lock on parent directory
• Session: commit_async() uses redo-log for memory extraction crash safety
• QueueFS: New SQLite backend (WAL + RecoverStale at-least-once)
• Documentation: Updated transaction mechanism docs (en/zh)
• Tests: 6 test files covering lock_context, lock_manager, path_lock, redo_log, concurrent_lock, and e2e

Recent Fixes

• Removed checkpoint dead code and fixed TOCTOU race condition in lock acquisition (fe33516)
• Fixed resource lock — add_resource finalize now acquires point lock on parent directory (715739d)
• Fixed path handling edge cases (1010bd4)

Testing

• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have tested this on the following platforms:
  • Linux
  • macOS
  • Windows

Checklist

• My code follows the project's coding style
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

Additional Notes

• 77 files changed, +3732/-1429 lines
• Design went through a major iteration: initial undo-based system replaced with lightweight lock + redo-log (commit 4e44a5d), reducing transaction code from ~4000 lines to ~700 lines

[CLAassistant]

All committers have signed the CLA.

[qin-ctx]

/review

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 390 - Partially compliant Compliant requirements: Transaction mechanism with journal and undo log implemented Rollback and crash recovery added Path locking with fencing tokens implemented Tests added for various scenarios Non-compliant requirements: (Cannot fully assess integration with all core operations without full diff) Requires further human verification: Integration with core operations (rm, mv, add_resource, session.commit) Multi-node deployment limitations and fallback behavior

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Potential Typo in Status Enum Verify that TransactionStatus.AQUIRE is the intended enum value (check for possible typo: ACQUIRE). elif tx.status in (TransactionStatus.INIT, TransactionStatus.AQUIRE): Async Method Calling Sync I/O Ensure that AGFS client calls (cat, write, ls, rm, stat) are properly awaited if they are async, or document that they are synchronous to avoid event loop blocking. content = self._agfs.cat(lock_path) if isinstance(content, bytes): return content.decode("utf-8").strip() return str(content).strip() except Exception: return None async def _is_locked_by_other(self, lock_path: str, transaction_id: str) -> bool: """Check if path is locked by another transaction (any lock type).""" token = self._read_token(lock_path) if token is None: return False lock_owner, _, _ = _parse_fencing_token(token) return lock_owner != transaction_id async def _create_lock_file( self, lock_path: str, transaction_id: str, lock_type: str = LOCK_TYPE_POINT ) -> None: """Create lock file with fencing token.""" token = _make_fencing_token(transaction_id, lock_type) self._agfs.write(lock_path, token.encode("utf-8"))

[MaojiaSheng]

related discussion in #472

[chuanbao666]

关于隔离级别是如何设计的？感觉像是Read Uncommitted？