Closed
Conversation
Emit STARTED status when beginning to read each stream and COMPLETE/INCOMPLETE terminal status when finishing. This is required by Airbyte protocol v0.2.0+ and fixes the error: "streams did not receive a terminal stream status message" Also update Dockerfile to use standard golang/alpine base images instead of pscale.dev private images.
Airbyte 2.x rejects LEGACY state messages with: IllegalArgumentException: LEGACY states are deprecated This changes the connector to emit one STATE message per stream with type=STREAM and a stream_descriptor, instead of a single global state blob. Each stream's shard cursors are emitted individually after the stream finishes processing.
Logger tests (cmd/internal/logger_test.go): - StreamState emits correct per-stream format with type=STREAM - StreamState handles multiple shards - No LEGACY "data" field present in state output - StreamStatus emits TRACE messages with correct status values - JSON round-trip produces exact Airbyte protocol v2 structure - Multiple stream states are independent Read protocol tests (cmd/airbyte-source/read_protocol_test.go): - Read emits per-stream STATE (not LEGACY global state) - STARTED and COMPLETE emitted for each stream - State descriptors have correct stream name and namespace - Message ordering: STARTED → STATE → COMPLETE - Multi-shard state contains all shard cursors - Read errors emit INCOMPLETE (not COMPLETE) and skip state emission
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.