Smart Pre-Processing Pipeline: Validation, Triage, and Corrections

[deucebucket]

Summary

A comprehensive pre-processing system that validates files, triages folders, and pushes corrections - all while preserving full automation.

Core philosophy: Don't trust garbage folders, DO trust audio. And if we screw up, fix it.

---

Status

Part	Feature	Status
1	File Validation	Done (merged in #179)
2	Folder Triage	This issue
3	Push Corrections	Split to separate issue (blocked on Skaldleita)

---

Part 2: Folder Triage (Local, Fast)

Categorize folders by "cleanliness" to decide processing strategy.

Detection patterns

Classify each folder as:

• Clean - Normal Author/Title structure, use path hints normally
• Messy - Scene tags, torrent markers, quality indicators - skip path parsing, trust audio only
• Garbage - Hash names, numbers only, generic folder names - skip path parsing, expect difficulty

Pattern examples

Category	Examples
Messy	{mb}, [FLAC], 64kbps, -TEAM suffix, .com in name
Garbage	a3f8b2c1d4e5, New Folder, tmp, numbers-only names

Processing strategy

Category	Path Parsing	Audio Processing	Confidence Modifier
Clean	Use as hints	Normal	None
Messy	SKIP	Audio-only	None
Garbage	SKIP	Audio-only	-10%

Implementation

• Add triage function with configurable patterns
• Integrate into scan pipeline (after file validation, before processing)
• Show triage results on dashboard (clean/messy/garbage counts)
• Setting: skip_messy_path_parsing (default: true)

---

Related

• Precog consensus voting: Implement consensus voting across data sources (Minority Report model) #102, PR feat: Precog consensus voting system (#102) #107
• Original discussion: [BUG] 3 last books stuck in the queue #79 (Merijeek's folder name observations)

Suggested by: @Merijeek (triage idea), @deucebucket (corrections concept)

[deucebucket]

Progress Update

Completed:

• ✅ Part 1: File Validation module (PR feat: File validation module (Part 1 of #110) #112) — ffprobe checks, duration/size sanity, truncation detection

Remaining:

• ⬜ Part 2: Folder Triage — pattern matching on folder names (clean/messy/garbage) to adjust processing strategy
• ⬜ Part 3: Push Corrections — Skaldleita-side corrections infrastructure + LM integration (requires SL API work first)