Skip to content

fix(stream): handle S3 upload errors without crashing the process#220

Merged
czirker merged 2 commits intomasterfrom
fix/ES-2976-s3-request-timeout-error-handling
Apr 2, 2026
Merged

fix(stream): handle S3 upload errors without crashing the process#220
czirker merged 2 commits intomasterfrom
fix/ES-2976-s3-request-timeout-error-handling

Conversation

@scotthawker
Copy link
Copy Markdown
Contributor

@scotthawker scotthawker commented Apr 2, 2026

Summary

  • Add error listener on internal S3 upload stream in leos3.js so emit("error") from toS3() has a handler and doesn't throw
  • Listen for both error and finish events in submitStream() with a callback guard, since _final(callback) with a truthy error emits error not finish

Task Reference

Changes Made

  • lib/stream/helper/leos3.js — Add .on('error') handler when creating the S3 stream in newStream(), and handle both error/finish events in submitStream()

Root Cause

When an S3 upload fails (e.g., RequestTimeout), toS3()'s .catch() calls toS3Stream.emit("error", err). In leos3.js, the internal s3 = ls.toS3(...) stream had no error listener, so emit("error") throws (Node.js EventEmitter behavior). Since the throw happens inside a .catch() handler, it creates a new unhandled promise rejection → Runtime.ExitError → Lambda crash.

Testing

  • Validated in staging against account 1000014534 which triggers this error every ~15 minutes

When an S3 upload fails (e.g., RequestTimeout), the error flows through
toS3()'s .catch() handler which calls toS3Stream.emit("error", err).
If no error listener is attached to that stream, emit("error") throws,
and since the throw happens inside a .catch() handler, it creates a new
unhandled promise rejection that crashes the Lambda runtime.

Three changes:

1. toS3() .catch(): Stop calling emit("error") on the stream. Just
   store the error and set the emittedError flag. The error propagates
   through the _final callback when the stream is ended.

2. leos3.js newStream(): Add an error listener on the internal s3
   upload stream so any error events have a listener and don't throw.

3. leos3.js submitStream(): Listen for both "error" and "finish"
   events with a callback guard. Previously only "finish" was handled,
   but _final(callback) with a truthy error emits "error" not "finish".

Task: ES-2976
Made-with: Cursor
@ch-snyk-sa
Copy link
Copy Markdown

ch-snyk-sa commented Apr 2, 2026

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues
Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@scotthawker scotthawker requested a review from czirker April 2, 2026 15:37
Revert the leo-stream.js change from 9f65041. The original behavior of
emitting the error on toS3Stream is correct — the fix in leos3.js adds
the missing error listener that catches it.

Task: ES-2976
@czirker czirker merged commit c8b53a5 into master Apr 2, 2026
8 checks passed
@czirker czirker deleted the fix/ES-2976-s3-request-timeout-error-handling branch April 2, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants