chore: ensure bizon raises an error when kafka connection fails#50
Closed
chore: ensure bizon raises an error when kafka connection fails#50
Conversation
* chore: remove python version condition * chore: pipeline should fail if source batch exceed full buffer size * fix test
* chore: updated streaming v2 to make it work * chore: added retry strategy * chore: remove all use_legacy_streaming_api params * chore: remove all use_legacy_streaming_api params
* chore: add custom dsm tracking * chore: add headers for metadata propagation * chore: fix when datadog isnt used * chore: add alias, refactor dsm monitoring call * chore: fix tests --------- Co-authored-by: Antoine Balliet <antoine.balliet@gorgias.com>
* chore: add return for DSM tracking * chore: add logs
* chore: loop through dsm trace * chore: typing * chore: typing
… config (#69) * chore: stop skipping message too large * chore: log consumer config * fix error message
* chore: change values and handle large rows in streaming v2 * chore: change values and handle large rows in streaming v2
* feat: add monitoring for large records synced in destination * fix import * fix * fix monitor in tests * fix test * fix * fix * fix * fix * add test for monitors * fix * fix
* chore: change values and handle large rows in streaming v2 * chore: change values and handle large rows in streaming v2 * chore: add monitoring trace * chore: remove service override * chore: add kafka headers * chore: add current span log * chore: wrap monitor to track iteration * chore: upgrade span priority * chore: remove priority * chore: set priority while removing kafka sampling prio * chore: auto keep sampling prio * chore: remove initial dd pathway * chore: add enable_tracing config
* feat: add destination_alias to sync_metadata * add unit testing
* chore: update config, add return when tracing disabled * chore: fix commented commit * chore: add forbidden extra fields
… logs (#76) * chore: removed logs * chore: instantiate monitor
…to BigQueryWrite client (#85) * chore: re-init client in append rows * chore: add kafka error handling, bizon api quota config * chore: handle source commit error at stream runer level * chore: add logs topic/destination_id on failure * chore: add debug logs * chore: remove unused instantion of BigQueryWriteClient
* chore: store all topic partitions manually * chore: production env check for commit
…(#88) * chore: update lockfile * chore: add invalid argument to retry exception list * chore: remove mix_stderr * chore: fix test
* chore: revert to commit consumer for all topics * remove manual offset tracking
… (#90) * feat: update avro deserializer to handle both 4 and 8 byte schema ids * chore: update gitignore * optimize code * Update .gitignore * fix message value * fix parameters --------- Co-authored-by: Anas El Mhamdi <anas.elmhamdi@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.