feat(transcoder): rewrite on libav (node-av) bindings#69
Open
devchaudhary24k wants to merge 4 commits intodevfrom
Open
feat(transcoder): rewrite on libav (node-av) bindings#69devchaudhary24k wants to merge 4 commits intodevfrom
devchaudhary24k wants to merge 4 commits intodevfrom
Conversation
Drop subprocess `spawn("ffmpeg")` in favor of in-process libav via
`node-av` (MIT, prebuilt binaries — no system ffmpeg needed).
- Decode-once / fan-out ABR pipeline: single demuxer + decoder feeds an
N-way split/asplit filter graph, one encoder+muxer per HLS rung.
Kills 2-3x duplicate decode work and the h264 threaded-decoder races
that flooded stderr with "reference picture missing" warnings.
- Poster: `yuv420p,setparams=range=pc` + `color_range: "pc"` to silence
deprecated pixel-format warning.
- Hover preview: output-packet-count stop condition instead of `trim`
filter to avoid EOF mid-pipeline.
- AAC tail drain via `encoder.encodeAll(null)` so the final 1024-sample
partial frame ships instead of being logged as "N frames left".
- Filtered libav log callback (`libav-log.ts`) suppresses four known
harmless h264 thread-race patterns; everything else flows through
with a `[libav]` prefix.
- BullMQ worker: `lockDuration` 10m / `lockRenewTime` 2m /
`stalledInterval` 60s — default 30s lock was flagging long transcodes
as stalled.
- Shared graceful-shutdown helper `@vidcastx/queue/shutdown` — SIGINT /
SIGTERM / SIGHUP → `worker.close(true)` so `tsx watch` restarts
release Redis locks cleanly instead of leaving zombies.
- Extended `workers/transcoder/ROADMAP.md` with HW-decoder design under
§4 and a new §8 for structured job logging; TODO summary at bottom.
Replaces the hardcoded (1080p/5000k, 720p/2800k, 480p/1400k) ladder with one scaled from a content-measured probe bitrate. Screencasts and talking heads stop paying for 5000k they don't need; high-motion content stops being starved. - `probeBitrate(inputPath, duration, fps)` runs libx264 `ultrafast` CRF=23 across 4 weighted sample windows (head 30s + 3×15s at 25%/55%/85% of source) and accumulates packet.size. Short-video fallback: <60s probes fully, <300s probes twice, >=300s uses the 4-sample scheme. Total probed footage is bounded (~75s) regardless of source length. - Output is discarded via the existing generator-chain pattern (encoder.packets → sum packet.size, no muxer). Same pattern as preview.ts. - `buildStreamVariants(height, fps, probeKbps)` scales per rung: 1080p = probe × 1.10, 720p × 0.55, 480p × 0.25, then fps multiplier. Clamped to [300 kbps, per-rung ceiling]. Rung-drop rule: remove any rung within 70% of the rung above. Result: screencasts can ship a single rung, vlogs 2-3, action content the full 3. - `runFFmpegTranscode` logs probe kbps + chosen ladder for observability; ROADMAP §1 checkboxes ticked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
spawn("ffmpeg")in favor of in-process libav vianode-av(MIT, prebuilt binaries — no system ffmpeg install required).split/asplitfilter graph; one encoder + HLS muxer per rung. Kills 2-3× duplicate decode work and the h264 threaded-decoder races that flooded stderr.@vidcastx/queue/shutdownhelper — SIGINT / SIGTERM / SIGHUP →worker.close(true)sotsx watchrestarts release Redis locks cleanly.lockDuration: 10m,lockRenewTime: 2m,stalledInterval: 60s— default 30s was flagging long transcodes as stalled.libav-log.ts) suppresses four known-harmless h264 thread-race patterns; everything else flows through with[libav]prefix.yuv420p,setparams=range=pc+color_range: "pc"to silence deprecated pixel-format warning.trimfilter (which EOF'd mid-pipeline and silently broke thumbnails).encoder.encodeAll(null)so the final 1024-sample partial frame ships instead of being dropped.workers/transcoder/ROADMAP.mdwith HW-decoder design under §4 and new §8 for structured job logging; TODO summary at bottom.db:studio --host=0.0.0.0, routeTree regen,IDEA.mdrewrite as product-vision doc.Test plan
pnpm check-types— clean across all 11 workspacespnpm --filter transcoder lint— cleanreference picture missingspam in stderrapps/appupload → transcoder →readystateprocessed/<org>/<vid>/master.m3u8+ per-rung playlists/segments +thumbnails/+previews/Notes
mainper request. Do not merge — leaving open for review.workers/transcoder/ROADMAP.mdtrack the remaining work beyond this PR.