release: beta.135 → beta.147 (includes critical data-safety fix #209)#212
Open
deucebucket wants to merge 20 commits intomainfrom
Open
release: beta.135 → beta.147 (includes critical data-safety fix #209)#212deucebucket wants to merge 20 commits intomainfrom
deucebucket wants to merge 20 commits intomainfrom
Conversation
…author names (#178) The "by Author" stripping regex now handles periods (F. Scott Fitzgerald), apostrophes (O'Brien), hyphens (Mary-Jane), and trailing parenthetical content like (Audiobook)(Nonfiction). Co-authored-by: deucebucket <deucebucket@users.noreply.github.com>
* Fix #110: Integrate file validation into scan pipeline Wire up the existing file_validation.py module into deep_scan_library() so invalid audio files (corrupt, truncated, too short) are caught before they enter the processing queue. Changes: - database.py: Add validation_status and validation_reason columns to books table - config.py: Add enable_file_validation, min_audio_duration_seconds, min_audio_file_size_mb settings - file_validation.py: Accept configurable min_duration and min_size overrides - app.py: Import and call validate_audio_file at all 4 queue insertion points (loose files, flat books, series books, regular books). Invalid files get status='validation_failed' and are excluded from the queue. ffprobe unavailability is non-blocking (validation skipped gracefully). - app.py: Add validation_failed counts to dashboard, /api/stats, and /api/library - database.py: Add validation_failed to should_requeue_book skip list - dashboard.html: Show validation failure alert when count > 0 - settings.html: Add File Validation toggle with configurable thresholds * chore: Bump version to beta.135 for file validation integration (#110) --------- Co-authored-by: deucebucket <deucebucket@users.noreply.github.com>
…180) Introduces the foundational registry that describes all processing layers, their ordering, enable/disable config keys, and circuit breaker dependencies. - LayerInfo: frozen dataclass with layer metadata (id, name, description, config key, default order, circuit breaker info) - LayerRegistry: lookup by id, ordered listing, enable/disable checks, custom pipeline_order support, order validation - default_registry: pre-populated with all 5 current layers matching the execution order in worker.process_all_queue Purely additive -- nothing uses the registry yet. No behavior changes. Co-authored-by: deucebucket <deucebucket@users.noreply.github.com>
…re flag (#182) Introduces a configurable PipelineOrchestrator that can replace the hardcoded process_all_queue sequence. Uses adapter pattern to wrap existing battle-tested layer functions with a uniform interface. - Add adapters.py: AudioIdAdapter, ApiLookupAdapter, AudioCreditsAdapter, AiVerifyAdapter, SlRequeueAdapter wrapping existing layer functions - Add orchestrator.py: PipelineOrchestrator handling layer ordering, circuit breaker waits, batch loops, rate limiting, stuck item recovery - Add use_modular_pipeline feature flag (default: False) for safe rollout - Update pipeline __init__.py with new exports When use_modular_pipeline is False (default), existing behavior is completely unchanged. When True, the orchestrator runs layers in the order defined by pipeline_order config using the registry and adapters. Co-authored-by: deucebucket <deucebucket@users.noreply.github.com>
* Addresses #66: Add pipeline configuration UI in settings * fix: Wire up pipeline layer toggles and add version bump/changelog - Add name attribute to pipeline layer toggle checkboxes so they submit with form - Bump version to beta.136 - Add CHANGELOG entry for pipeline configuration UI * fix: Use existing json import instead of inline alias --------- Co-authored-by: deucebucket <deucebucket@users.noreply.github.com>
* Addresses #66: Add standalone layer execution API and UI Adds POST /api/pipeline/run-layer/<layer_id> endpoint that runs a single pipeline layer on demand (synchronous). Adds a play button next to each layer in the settings pipeline section with spinner feedback and result badges showing processed/resolved counts. * chore: Bump version to beta.137 for standalone layer execution (#66) --------- Co-authored-by: deucebucket <deucebucket@users.noreply.github.com>
Adds Plugins tab to Settings with 4-step wizard for no-code custom API layers. Includes plugins Blueprint with CRUD + test endpoints, wizard modal (info, API config, response mapping, test & save), and full integration with existing patterns. Closes #186
Adds Python drop-in plugin system. Users drop plugin directories into /data/plugins/ with manifest.json and a BasePlugin subclass. Features: auto-discovery via importlib, PluginAdapter for pipeline compatibility, timeout enforcement, deep copy isolation, exception handling, metrics recording with auto-disable, and an example plugin template. Closes #188
…ttings reorg (#198) (#199) Professional UI overhaul: extracted 730 lines CSS to static/css/style.css with design tokens, consolidated duplicate JS to static/js/common.js, reorganized settings from 7 tabs to 4 (Library, Engine, Pipeline, Integrations), fixed color palette, added mobile responsive breakpoints, replaced all hardcoded colors with CSS variables. Closes #198
* Fix #201: Resolve path normalization mismatch causing SAFETY BLOCK for library root files Windows mapped drives (e.g. R:\) resolve to UNC paths (\\server\share) but library path comparisons in the processing layers used raw config paths without .resolve(), causing the match to fail. The fallback then used parent.parent which went above the library root for loose files. Fixed all 4 locations to resolve both sides before comparison, and added smart fallback that checks if the path is a file before deciding depth. * Bump APP_VERSION to beta.144, update README badge and CHANGELOG * Fix same path resolution bug in layer_content.py Missed 5th location with identical unresolved relative_to() and parent.parent fallback — caught by vibe-check review. --------- Co-authored-by: deucebucket <deucebucket@users.noreply.github.com>
* Docs #203: Plugin system documentation and discoverability Add in-app docs for Python drop-in plugins, secrets management, and real-world API examples (Google Books, Open Library) to the Plugins settings tab. Ship example-logger plugin to examples/plugins/ with comprehensive README. * Fix extra brace in Google Books URL template example --------- Co-authored-by: deucebucket <deucebucket@users.noreply.github.com>
…ttings toggle (#206) Backend folder triage (clean/messy/garbage classification) was already implemented but invisible in the UI. Now surfaces triage data: - Dashboard info banner showing messy/garbage folder counts - Library view badges on affected books (Messy/Garbage indicators) - Settings toggle to enable/disable folder triage - Triage data in "all" library API responses Split push corrections to #205 (blocked on Skaldleita backend). Co-authored-by: deucebucket <deucebucket@users.noreply.github.com>
When an issue gets the 'ecosystem' label and references another deucebucket repo (e.g. deucebucket/skaldleita#127), automatically comments on the partner issue with a cross-reference link. Both repos have the same workflow and share an ECOSYSTEM_PAT secret.
When 'Use hard links' was enabled and the watch folder and library were on different filesystems, os.link() raised EXDEV and the code silently fell back to shutil.copy2() then deleted the originals. That broke torrent seeding, doubled disk use, and silently violated the user's explicit preference. - Add filesystem-compatibility pre-check in move_to_output_folder when use_hard_links is True. Returns a clear, actionable error if source and library are on different st_dev, without touching any source files. - Remove the EXDEV copy+delete fallback from both the single-file and directory-loop branches. Remaining OSErrors (permission, ENOSPC, etc.) propagate to the outer handler with source intact; the watch worker records the failure as watch_folder_error visible in the UI. - Bump APP_VERSION to 0.9.0-beta.147 and update README badge + CHANGELOG. Reported by @kyleviloria. Co-authored-by: deucebucket <deucebucket@users.noreply.github.com>
…tice (#213) Problem 1: watch_folder_processed was an in-memory set() that got wiped on every restart. Any file that couldn't be processed (unknown author, ambiguous match, move failure, mtime churn) got re-submitted every scan cycle after a restart, forever. One LM instance generated ~48% of all Skaldleita /match traffic for days on the same filename. Problem 2: Skaldleita PR #129 added a server_notice field to /match responses. When the server detects a retry loop it now sends {severity, code, message, action: abort_task, upgrade_url}. LM ignored it. Fixes: - New watch_folder_processed SQLite table (path PK, processed_at, outcome, error_message). outcome in {moved, move_failed, aborted_by_server}. - watch_folder_is_processed() / watch_folder_mark_processed() helpers in library_manager/database.py. process_watch_folder swapped from set ops to these helpers. Restart no longer resets dedup. - bookdb.py logs every server_notice (with upgrade_url). On action=abort_task it stashes the notice in a threading.local() slot so scope is per-thread (watch worker, API endpoint, pipeline layer don't cross-contaminate). - process_watch_folder reads the abort slot after each identify attempt; if set, marks the item as aborted_by_server and skips the pipeline. Bumps APP_VERSION to 0.9.0-beta.148. Co-authored-by: deucebucket <deucebucket@users.noreply.github.com>
![bucket-agent[bot]](https://avatars.githubusercontent.com/in/2751175?s=60&v=4){kind=small}
There was a problem hiding this comment.
🔍 Vibe Check Review
Context
Release PR aggregating beta.135 → beta.148 from develop into main. The headline is a critical data-safety fix (#209: stop copy+delete when hardlinking across filesystems) plus a watch-folder retry-loop fix (#208). Most other changes (modular pipeline, plugin system, custom layer builder, UI overhaul) have already been individually reviewed and merged to develop.
Codebase Patterns I Verified
library_manager/database.py:153usesALTER TABLE ... ADD COLUMNinsidetry/except: passfor idempotent schema migrations — the newvalidation_status/validation_reasoncolumns match this pattern.move_to_output_folderatapp.py:6500uses(success, new_path, error)tuple return — newst_devmismatch error path conforms._abort_state = threading.local()inbookdb.py:40— consumer atapp.py:6836-6842clears after every identify attempt, so no leak across watch items.use_modular_pipelineflag defaultsFalseinconfig.py:121, gated inworker.py:168— zero behavior change on merge.- CHANGELOG.md (Keep-a-Changelog) updated through beta.148; README version badge bumped 134→148.
✅ Good
- Critical fix is correct:
app.py:6527-6538pre-checksst_devmatch before any file operation; the dangerousused_copy_fallback/files_to_deletecodepath is fully removed — sources are now guaranteed untouched on cross-filesystem hardlink failure. - Empty-folder cleanup guard is right:
app.py:6650now only runs whennot use_hard_links— no more accidental source removal when originals were meant to stay. - Error message is actionable: tells the user exactly how to fix it ("Move your library to the same volume... or disable 'Use hard links' in Settings"), confirms source files were not modified.
- #208 retry-loop fix is thorough: SQLite-backed dedup (
watch_folder_processed) survives restarts, and theserver_notice/abort_taskintegration gives Skaldleita a clean kill switch without layering more timeout heuristics in LM.
🚨 Issues Found
No issues found.
📋 Scope Verification
| Issue | Problem | Addressed? | Notes |
|---|---|---|---|
| #209 | Hardlink failure silently copies+deletes originals | ✅ | app.py:6524-6538 pre-check + fallback removed at 6624-6627, 6644-6645 |
| #208 (implicit via beta.148) | In-memory watch dedup wiped on restart; Skaldleita retry-loop hammer | ✅ | database.py:182 table + helpers; bookdb.py:40 notice handler; app.py:6461/6836/6841/6896/6931 integration |
| #66, #110, #169, #178–#199, #201–#206 | Prior develop merges | ✅ | All already reviewed and merged to develop; this PR only aggregates them for release |
Scope Status: SCOPE_OK
📝 Documentation Check
- CHANGELOG.md: ✅ Updated (beta.147 + beta.148 entries with issue refs and fix details)
- README.md: ✅ Updated (version badge 134→148, "Recent Changes" section extended through beta.148)
🎯 Verdict
APPROVE — The critical data-safety fix is implemented correctly (pre-check + full removal of the destructive fallback), the #208 persistence fix is sound, and every other change in the bundle has already been individually reviewed on its way into develop. Ship it.
… column (#214) Three INSERT statements in process_watch_folder targeted a column 'added_at' on the books table. The real column is 'created_at' — 'added_at' only exists on the queue table. Every watch-folder INSERT raised sqlite3.OperationalError which two separate except blocks caught with logger.debug(), hiding the error at default log levels. Effects: - Successful watch-folder moves never produced a 'pending' or 'needs_attention' row in the books table. - Failed watch-folder moves never produced a 'watch_folder_error' row — the UI had no surface for the failure; only the log showed it. Fixes: - Renamed 'added_at' to 'created_at' in the three books-table INSERTs at app.py:6906, 6914, 6944. - Raised both swallow-except blocks from logger.debug to logger.warning with exc_info=True so future failures don't rot silently. Surfaced during live testing of the #209 fix. Bug has existed since the watch-folder feature was introduced. Bumps APP_VERSION to 0.9.0-beta.149. Co-authored-by: deucebucket <deucebucket@users.noreply.github.com>
There was a problem hiding this comment.
🔍 Vibe Check Review
Context
Release PR rolling up beta.135 → beta.149 from develop → main. Includes the modular pipeline architecture (#66 family), plugin system + health dashboard, UI overhaul, folder triage UI, and three data-safety bug fixes (#209 hardlink EXDEV, #208 watch-folder retry storm, #211 phantom added_at column).
Codebase Patterns I Verified
- Hardlink fix (#209) is intact:
app.py:6524-6538has thest_devpre-check; grepping confirmed noshutil.copy2remains as an EXDEV fallback inmove_to_output_folder— the only remainingshutil.copy2in the file is at line 12077 (unrelated backup code). - CHANGELOG ordering: new entries added at top in Keep-a-Changelog style matching existing format.
- Sub-PRs were individually reviewed and approved into develop (commits
7864dcf,6cd29e3,24f0888, etc. visible on develop); this PR is the merge cut, not net-new code.
✅ Good
- Critical data-safety fix (#209) verified end-to-end: pre-check present, fallback removed, source files untouched on EXDEV.
- Two bonus data-safety fixes landed beyond what the title advertises (#208, #211), both making previously-silent failures loud.
- CHANGELOG + README both updated; release notes describe behavior-changing fixes accurately.
🚨 Issues Found
No issues found.
📋 Scope Verification
Release rollup — all linked issues were addressed in their individual sub-PRs prior to this merge. Spot-checked #209 (hardlink) against current app.py and confirmed the fix shipped.
Scope Status: SCOPE_OK
Minor note (non-blocking): PR title says "beta.135 → beta.147" but the actual diff includes beta.148 and beta.149 (commits 6cd29e3 and 24f0888 landed on develop after the PR was opened). Consider updating the title to "beta.135 → beta.149" before merge for accurate release tagging. Not a blocker — content is all additive data-safety work.
📝 Documentation Check
- CHANGELOG.md: ✅ Updated (new entries for 147/148/149)
- README.md: ✅ Updated
🎯 Verdict
APPROVE
Ship it. All sub-PRs were already reviewed; the critical data-safety fix is verified present in the merged state; CHANGELOG/README are current. Only suggestion is cosmetic: update the PR title to reflect beta.149 as the final version before cutting the tag.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release: 0.9.0-beta.147
Promotes 18 commits from `develop` to `main`. Ships as `:latest` Docker tag on merge.
#209 — Hard link failure silently copies and deletes originals. Reported by @kyleviloria on beta.134 (the current `:latest`). When `Use hard links` was enabled and the watch folder / library lived on different filesystems, `os.link()` raised `EXDEV` and LM silently fell back to `shutil.copy2()` + deleting each original file — destroying torrent seeds, doubling disk use, and silently violating the user's explicit preference. Fixed in #210 with an `st_dev` pre-check that fails fast with a clear message and leaves source files untouched. Live-tested against the maintainer's 1193-book library across 3 scenarios (cross-fs, same-fs, no-hardlinks) — all passed.
Features shipped
Fixes shipped
Full commit list
```
7864dcf Fix #209: Hard link failure no longer copies+deletes originals (#210)
d47e916 Add ecosystem cross-repo sync workflow
1bd11bf Feat #110: Add folder triage UI - dashboard stats, library badges, settings toggle (#206)
a06840c docs: Plugin system documentation and discoverability (#203) (#204)
b6f6a9d fix: Resolve SAFETY BLOCK for files in library root (#201) (#202)
2986cdb feat: Professional UI overhaul - design system, CSS/JS extraction, settings reorg (#198) (#199)
3c973c1 feat: Add drop-in Python plugin system (#188) (#197)
c246f37 feat: Add Plugin Health Dashboard with auto-disable circuit breaker (#189) (#196)
3a63024 feat: Add Custom Layer Builder wizard UI (#186) (#195)
19a4d1d feat: Expand hooks with event system and body templates (#187) (#194)
6398849 feat: Template HTTP layers - no-code custom API sources (#185) (#191)
d0e07b3 feat: Add standalone layer execution API and UI (#66) (#184)
6182293 feat: Add pipeline configuration UI in settings (#66) (#183)
a09343b Addresses #66: Add PipelineOrchestrator with layer adapters and feature flag (#182)
04d625e Addresses #66: Add pipeline_order to default config (#181)
8072c18 Addresses #66: Add LayerRegistry and LayerInfo for modular pipeline (#180)
d51dac5 feat: Integrate file validation into scan pipeline (#110) (#179)
3699f9c Fix #169: Expand clean_search_title regex to handle special chars in author names (#178)
```
Test plan
Merge strategy
Use `Create a merge commit` (not squash). Release PRs to main preserve full history per project convention — individual fix/feature commits stay attributable.