Skip to content

[BUG] Watch-folder worker contends with library scan, hits 'database is locked' timeouts #215

@deucebucket

Description

@deucebucket

What happened

Surfaced live while testing #211 on beta.149. When the library-scan worker is mid-transaction (e.g. walking `/mnt/rag_data/audiobooks/` and processing many series folders), any call the watch-folder worker makes through `watch_folder_mark_processed` (which opens its own short-lived connection in `library_manager/database.py`) waits the full 30-second `busy_timeout` and then fails with:

```
ERROR - Watch folder: Error processing /mnt/.../Test Author - 211 Happy: database is locked
```

The watch-folder worker's outer `try/except` catches it and returns `Processed 0/1 items`. On the next scan, the item is picked up again — eventually processed correctly when scan is idle, but wasteful until then.

Reproducible: during a live scan cycle (just after restart, with a real library attached), drop a new file in the watch folder. ~30s lock timeout → `database is locked` error logged.

Root cause

Two workers open separate SQLite connections to the same DB and write concurrently:

WAL mode is enabled and `busy_timeout=30000` is set — that usually absorbs small overlaps — but a long scan layer transaction keeps the write lock held past the 30s cap.

Options to consider

  1. Route `watch_folder_mark_processed` through the existing watch-worker connection instead of opening a new one. The watch worker already has `conn = get_db()` at `app.py:6700` — we could pass it (or the cursor) into the helper. Avoids the second writer entirely for the watch path.
  2. Raise the busy_timeout for the watch path only (e.g. 60-90s). Simple but hides the underlying contention.
  3. Split watch-folder state into its own DB file (`watch.db`). Strongest isolation but requires more plumbing.
  4. Chunk the scan worker's transactions so the lock is held for shorter windows. Broader change with its own tradeoffs.

Option 1 is probably the smallest and most correct — the watch worker owns the transaction lifetime for its item and already commits at the end.

Impact (current, before fix)

  • Low-severity: watch items are eventually processed once the scan quiets down.
  • Higher-severity when the scan runs continuously (e.g. during initial import of a big library) — watch items could loop for minutes before succeeding. Combined with [BUG] watch_folder_processed is in-memory only + add handler for Skaldleita server_notice #208's persistent dedup, a watch item could get marked `move_failed` by mistake when the lock-timed-out failure bubbles out via the outer `except`.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions