You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Surfaced live while testing #211 on beta.149. When the library-scan worker is mid-transaction (e.g. walking `/mnt/rag_data/audiobooks/` and processing many series folders), any call the watch-folder worker makes through `watch_folder_mark_processed` (which opens its own short-lived connection in `library_manager/database.py`) waits the full 30-second `busy_timeout` and then fails with:
The watch-folder worker's outer `try/except` catches it and returns `Processed 0/1 items`. On the next scan, the item is picked up again — eventually processed correctly when scan is idle, but wasteful until then.
Reproducible: during a live scan cycle (just after restart, with a real library attached), drop a new file in the watch folder. ~30s lock timeout → `database is locked` error logged.
Root cause
Two workers open separate SQLite connections to the same DB and write concurrently:
Scan worker (in `process_all_queue` / layer pipeline) holds write transactions for the length of each layer, sometimes tens of seconds.
WAL mode is enabled and `busy_timeout=30000` is set — that usually absorbs small overlaps — but a long scan layer transaction keeps the write lock held past the 30s cap.
Options to consider
Route `watch_folder_mark_processed` through the existing watch-worker connection instead of opening a new one. The watch worker already has `conn = get_db()` at `app.py:6700` — we could pass it (or the cursor) into the helper. Avoids the second writer entirely for the watch path.
Raise the busy_timeout for the watch path only (e.g. 60-90s). Simple but hides the underlying contention.
Split watch-folder state into its own DB file (`watch.db`). Strongest isolation but requires more plumbing.
Chunk the scan worker's transactions so the lock is held for shorter windows. Broader change with its own tradeoffs.
Option 1 is probably the smallest and most correct — the watch worker owns the transaction lifetime for its item and already commits at the end.
Impact (current, before fix)
Low-severity: watch items are eventually processed once the scan quiets down.
Higher-severity when the scan runs continuously (e.g. during initial import of a big library) — watch items could loop for minutes before succeeding. Combined with [BUG] watch_folder_processed is in-memory only + add handler for Skaldleita server_notice #208's persistent dedup, a watch item could get marked `move_failed` by mistake when the lock-timed-out failure bubbles out via the outer `except`.
What happened
Surfaced live while testing #211 on beta.149. When the library-scan worker is mid-transaction (e.g. walking `/mnt/rag_data/audiobooks/` and processing many series folders), any call the watch-folder worker makes through `watch_folder_mark_processed` (which opens its own short-lived connection in `library_manager/database.py`) waits the full 30-second `busy_timeout` and then fails with:
```
ERROR - Watch folder: Error processing /mnt/.../Test Author - 211 Happy: database is locked
```
The watch-folder worker's outer `try/except` catches it and returns `Processed 0/1 items`. On the next scan, the item is picked up again — eventually processed correctly when scan is idle, but wasteful until then.
Reproducible: during a live scan cycle (just after restart, with a real library attached), drop a new file in the watch folder. ~30s lock timeout → `database is locked` error logged.
Root cause
Two workers open separate SQLite connections to the same DB and write concurrently:
WAL mode is enabled and `busy_timeout=30000` is set — that usually absorbs small overlaps — but a long scan layer transaction keeps the write lock held past the 30s cap.
Options to consider
Option 1 is probably the smallest and most correct — the watch worker owns the transaction lifetime for its item and already commits at the end.
Impact (current, before fix)
Related