Feat: NZB Preloading & Failed NZB Management

[docloulou]

NZB Preloading & Failed NZB Management

🎯 Overview

This PR introduces three major improvements for Usenet streaming services (NzbDAV/Altmount/Torbox):

1. NZB Preloading: Automatically send top X NZBs to your usenet service before you click, so downloads start immediately
2. Smart Conflict Resolution: Handle concurrent preload/resolve operations gracefully with WebDAV polling
3. Failed NZB Filtering: Properly detect and filter NZBs that have failed in SABnzbd

✨ Features

1. NZB Preloading

Global Settings (Miscellaneous)

• Enable/disable preloading globally
• Configure number of NZBs to preload (1-20, default: 3)
• Fire-and-forget async operation (non-blocking)

Per-Addon Control (Newznab Settings)

• Enable preloading per newznab indexer
• Default: OFF (opt-in per indexer)
• Only streams from enabled indexers are preloaded

How it works:

1. After sorting/filtering streams, the top X usenet streams are selected
2. NZBs are sent to NzbDAV/Altmount/Torbox via addUrl() in the background
3. Preloading happens AFTER the response is sent to the user
4. Downloads start immediately, ready when the user clicks

2. Smart Conflict Resolution

Problem Solved:
When a user clicks on a stream that's currently being preloaded, addUrl() would fail with an internal server error (duplicate NZB).

Solution:

• Track which NZB URLs are currently being preloaded
• If addUrl() fails during resolve and the NZB is in the preload tracker:
  • Poll WebDAV every 1 second (up to 80s timeout)
  • Wait for the content directory to appear
  • Continue normally once the preload completes
• If not a preload, fail immediately (preserve original error behavior)

3. Failed NZB Management

Problem Solved:
NZBs that failed in SABnzbd without creating a WebDAV folder were not properly detected, appearing in results even though they're unavailable.

Solution:

• listNzbs() now includes failed entries from SABnzbd history even without WebDAV folders
• Per-service toggle: "Filter Failed NZBs" (default: enabled)
• Available in NzbDAV and Altmount credentials settings
• When disabled, failed NZBs appear in results (useful for debugging)

📁 Modified Files

Core (packages/core/src/)

File	Changes
db/schemas.ts	Added preloadNzb to UserDataSchema Changed ServiceSchema.credentials from z.string() to z.any()
debrid/base.ts	Added optional preloadNzb() method to DebridService interface
debrid/usenet-stream-base.ts	• Implemented preloadNzb() with WebDAV check + addUrl() • Added preload tracking (preloadingUrls Set) • Conflict resolution: WebDAV polling on addUrl failure • Fixed listNzbs() to include failed SABnzbd entries
debrid/utils.ts	Added filterFailedNzbs?: boolean to BuiltinDebridServices
main.ts	• Added _preloadNzbs() fire-and-forget pipeline • Filters by addon.preset.options.preloadNzb === true • Runs after _processStreams via setImmediate
presets/newznab.ts	Added "Preload NZB" option (boolean, default: false, advanced)
presets/builtin.ts	Modified getBaseConfig() to pass filterFailedNzbs from service credentials
builtins/utils/debrid.ts	Conditional filtering of failed NZBs based on service.filterFailedNzbs
utils/constants.ts	Added "Filter Failed NZBs" credential option to NzbDAV and Altmount

Frontend (packages/frontend/src/)

File	Changes
components/menu/miscellaneous.tsx	Added "Preload NZB" settings card with Switch + NumberInput
components/menu/services.tsx	Fixed boolean value handling in service credentials modal

🔧 Configuration

For End Users

Step 1: Enable Global Preloading

1. Navigate to Settings → Miscellaneous
2. Find "Preload NZB" section
3. Toggle Enable ON
4. Set Number of NZBs to preload (1-20, recommended: 3-5)

Step 2: Enable Per-Indexer

1. Navigate to Settings → Addons
2. Open settings for your Newznab indexer (e.g., NZBgeek)
3. Scroll to "Preload NZB" (in advanced settings)
4. Toggle ON

Step 3: Configure Failed NZB Filtering (Optional)

1. Navigate to Settings → Services
2. Click settings icon for NzbDAV or Altmount
3. Find "Filter Failed NZBs" toggle
4. Default: ON (recommended)
5. Turn OFF only for debugging purposes

For Developers

Preload Flow:

User Request → getStreams() → _processStreams() → Return Response
                                       ↓ (fire-and-forget)
                                _preloadNzbs() → preloadNzb() → addUrl()

Resolve Flow (with preload conflict):

User Clicks → resolve() → addUrl() [FAILS: duplicate]
                              ↓
                      Check preloadingUrls
                              ↓
                      Poll WebDAV (1s intervals)
                              ↓
                      Content appears → Continue

🧪 Testing

only tested with NZBDAV

Test Preloading

1. Enable preloading globally (count: 3)
2. Enable preloading on a Newznab addon
3. Search for a movie/series with many NZB results
4. Check NzbDAV logs: you should see Preloaded NZB entries
5. Check SABnzbd queue: top 3 NZBs should be downloading

Test Conflict Resolution

1. Enable preloading (count: 5)
2. Search for content with NZB results
3. Immediately click on one of the top 5 streams (before preload completes)
4. Expected: Stream starts after a few seconds (waits for preload to finish)
5. Should NOT see "internal server error" or "failed to queue NZB"

Test Failed NZB Filtering

1. Manually fail an NZB in SABnzbd (or let one fail naturally)
2. Search for that content
3. With "Filter Failed NZBs" enabled: failed NZB should NOT appear
4. Disable the toggle in service settings
5. Search again: failed NZB should now appear in results

🐛 Known Limitations

• Preloading only works for NzbDAV and Altmount (usenet streaming services)
• Other services like Stremio NNTP or Easynews are not affected
• Preload tracking is in-memory only (lost on server restart)
• WebDAV polling timeout is 80 seconds (hard-coded)
• Torbox have very very low api rate limit so 429 is reach very easily during my test

🔄 Breaking Changes

None. All features are opt-in and backward compatible:

• Preloading is OFF by default (both globally and per-addon)
• Failed NZB filtering is ON by default (existing behavior)
• ServiceSchema.credentials now accepts any type (was string-only) but remains compatible

📊 Performance Impact

• Preloading: Minimal overhead, runs asynchronously after response
• Conflict Resolution: Only triggered when clicking on a preloading NZB (rare)
• Failed NZB Detection: One-time cost during listNzbs() (already cached)

🙏 Acknowledgments

This feature addresses the long-standing issue of slow usenet stream startup times by proactively downloading the most likely streams. The conflict resolution ensures a smooth user experience even when clicking quickly.

---

Ready to merge ✅

Summary by CodeRabbit

New Features

• Added NZB preloading feature with enable/disable toggle in settings
• Added count configuration for number of NZBs to preload (1–20)
• Added addon filter selection for preloading across supported debrid services
• Implemented proactive NZB preloading with WebDAV content verification and automatic retry logic

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

A new NZB preloading feature is introduced across the entire stack. It adds user preference storage for preload configuration, defines a debrid service interface method, implements preloading logic in multiple services with concurrent request tracking and WebDAV polling, orchestrates asynchronous preloading within the main stream resolution flow, and provides UI controls for enabling and configuring preload behaviour.

Changes

Cohort / File(s)	Summary
Data Schema packages/core/src/db/schemas.ts	Added optional preloadNzb field to UserDataSchema with enabled flag, count limit (1–20), and addon array.
Service Interface packages/core/src/debrid/base.ts	Added optional preloadNzb(nzbUrl, category, expectedFolderName) method to DebridService interface.
Service Implementations packages/core/src/debrid/usenet-stream-base.ts, packages/core/src/debrid/torbox.ts	Implemented preloadNzb with per-service tracking (static Set), WebDAV existence checks, history-based skip logic, polling on resolution failures, and error handling.
Orchestration Logic packages/core/src/main.ts	Added fire-and-forget _preloadNzbs method that selects top NZB-bearing streams, encodes per-service credentials, and asynchronously preloads via debrid services after getStreams response.
Frontend UI packages/frontend/src/components/menu/miscellaneous.tsx	Added Preload NZB SettingsCard with Switch for enable/disable, NumberInput for count (1–20, default 3), and Combobox for addon filtering from presets.
Configuration packages/core/src/utils/config.ts	Minor whitespace and indentation alignment; no functional changes.

Sequence Diagram

sequenceDiagram
    actor User
    participant UI as Frontend UI
    participant Main as getStreams (Main)
    participant Orchestrator as _preloadNzbs
    participant Services as Debrid Services
    participant WebDAV as WebDAV Storage

    User->>UI: Enable preload & configure
    UI->>Main: Call getStreams()
    Main->>Main: Prepare stream response
    Main->>Orchestrator: Fire _preloadNzbs(streams, context)
    Note over Orchestrator: Async (fire-and-forget)
    Orchestrator->>Orchestrator: Select top NZB streams
    Orchestrator->>Orchestrator: Encode credentials per service
    loop For each stream
        Orchestrator->>Services: preloadNzb(url, category, folderName)
        Services->>Services: Check existing downloads
        alt Already exists
            Services->>Orchestrator: Skip (log)
        else Not present
            Services->>Services: Call addNzb()
            Services->>WebDAV: Poll for content
            WebDAV-->>Services: Content appears/timeout
            Services->>Orchestrator: Success/error (logged)
        end
    end
    Orchestrator-->>Main: Complete (gracefully)
    Main-->>User: Return streams

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested reviewers

• Viren070

Poem

🐰 Whiskers aquiver with delight,
NZBs preload through the night,
WebDAV checks and tokens dance,
Per-service calls in async trance,
Fire-and-forget, oh what a treat! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Merge Conflict Detection	⚠️ Warning	❌ Merge conflicts detected (10 files): ⚔️ packages/core/src/db/schemas.ts (content) ⚔️ packages/core/src/debrid/base.ts (content) ⚔️ packages/core/src/debrid/torbox.ts (content) ⚔️ packages/core/src/debrid/usenet-stream-base.ts (content) ⚔️ packages/core/src/formatters/base.ts (content) ⚔️ packages/core/src/main.ts (content) ⚔️ packages/core/src/metadata/tvdb.ts (content) ⚔️ packages/core/src/utils/config.ts (content) ⚔️ packages/frontend/src/components/menu/miscellaneous.tsx (content) ⚔️ packages/frontend/src/components/menu/save-install.tsx (content) These conflicts must be resolved before merging into main.	Resolve conflicts locally and push changes to this branch.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately reflects the main changes: NZB preloading functionality and failed NZB management are clearly the primary features added across multiple files.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

⚔️ Resolve merge conflicts (beta)

• Auto-commit resolved conflicts to branch add-preload-nzb
• Post resolved changes as copyable diffs in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🤖 Fix all issues with AI agents

In `@packages/core/src/debrid/usenet-stream-base.ts`:
- Around line 430-431: The static Set UsenetStreamService.preloadingUrls
currently stores raw NZB URLs globally causing cross-service waits and unbounded
growth; change keys to be service-scoped (e.g., combine service identifier with
the NZB URL into a preloadKey used wherever preloadingUrls is referenced),
remove entries when content already exists and immediately after a successful
resolve (delete preloadKey in the branches handling "content already exists" and
after "contentPath confirmed"), and implement a TTL cleanup (store timestamps or
use a Map<preloadKey, timestamp> and periodically purge entries older than the
TTL) to handle preloads that never resolve; update all usages of preloadingUrls
(including in addUrl and resolve code paths mentioned around
UsenetStreamService.preloadingUrls and lines ~600–625 and ~942–1001) to use the
new scoped key logic and removal behavior.
- Around line 627-632: The debug log in serviceLogger.debug is leaking raw NZB
URLs (nzbUrl) which may contain API keys; replace the raw nzbUrl with a masked
version before logging—e.g., call an existing masking helper (maskNzbUrl or
similar) to produce maskedNzbUrl = maskNzbUrl(nzbUrl) (or implement a small
maskUrl helper that strips query params/credentials) and use maskedNzbUrl in the
Preloaded NZB debug call while keeping category, expectedFolderName, and nzoId
unchanged.

In `@packages/core/src/main.ts`:
- Around line 2493-2509: The folder name is being derived from stream.url which
can be a proxied download URL; for NZB-type streams use stream.nzbUrl (falling
back to stream.filename) so the preload folder matches getExpectedFolderName()
used during resolve; update the logic around folderName (the block that
currently reads stream.url and sets folderName) to prefer stream.nzbUrl when
stream.type indicates an NZB (or when stream.nzbUrl exists), still URL-decode
the last path segment like decodeURIComponent(new
URL(...).pathname.split('/').pop()), and keep the existing fallback to
stream.filename ?? 'unknown_nzb'.

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@packages/core/src/debrid/torbox.ts`:
- Around line 598-618: The logged nzbUrl in the TorBox preload path may contain
indexer API keys; update the calls that log nzbUrl (the logger.debug in the "NZB
already exists" branch, the catch branch, and the success log after await
this.addNzb) to use a redacted version instead of the raw URL: implement a small
helper (e.g., maskNzbUrl or redactNzbUrl) that strips or masks sensitive query
params (like api_key, apikey, key, token) or otherwise replaces the query string
with a placeholder, and pass that masked value into the logger.debug metadata
(while leaving nzbUrl unchanged for actual operations such as this.addNzb).

[docloulou]

Ready to be reviewed @Viren070