forward: enable skip_invalid_event by default #587
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
|
Thanks! |
daipom
pushed a commit
to fluent/fluentd
that referenced
this pull request
Jun 26, 2025
**Which issue(s) this PR fixes**: Fixes # **What this PR does / why we need it**: This PR will enable `skip_invalid_event` in `in_forward` by default. If broken data is received for any reason, it may cause the following error. ``` 2025-06-12 18:27:30 +0900 [error]: unexpected error on reading data host="127.0.0.1" port=55970 error_class=TypeError error="can't convert String into an exact number" 2025-06-12 18:27:30 +0900 [error]: <internal:timev>:325:in 'Time.at' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/formatter_stdout.rb:42:in 'Fluent::Plugin::StdoutFormatter#format' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/out_stdout.rb:67:in 'Fluent::Plugin::StdoutOutput#format' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/out_stdout.rb:60:in 'block in Fluent::Plugin::StdoutOutput#process' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/event.rb:259:in 'block in Fluent::MessagePackEventStream#each' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/event.rb:258:in 'Array#each' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/event.rb:258:in 'Enumerable#each_with_index' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/event.rb:258:in 'Fluent::MessagePackEventStream#each' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/out_stdout.rb:59:in 'Fluent::Plugin::StdoutOutput#process' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/output.rb:865:in 'emit_sync' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/event_router.rb:115:in 'Fluent::EventRouter#emit_stream' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/in_forward.rb:329:in 'Fluent::Plugin::ForwardInput#on_message' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/in_forward.rb:226:in 'block in Fluent::Plugin::ForwardInput#handle_connection' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/in_forward.rb:263:in 'block (3 levels) in Fluent::Plugin::ForwardInput#read_messages' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/in_forward.rb:262:in 'MessagePack::Unpacker#feed_each' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/in_forward.rb:262:in 'block (2 levels) in Fluent::Plugin::ForwardInput#read_messages' 2025-06-12 18:27:30 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/in_forward.rb:273:in 'block in Fluent::Plugin::ForwardInput#read_messages' ... ``` When enable `skip_invalid_event`, the broken data are just skipped. ``` 2025-06-18 11:16:40 +0900 [info]: #0 fluentd worker is now running worker=0 2025-06-18 11:16:49 +0900 [warn]: #0 skip invalid event: host="127.0.0.1" tag="incoming" time=-17 record=nil 2025-06-18 11:16:49 +0900 [warn]: #0 skip invalid event: host="127.0.0.1" tag="incoming" time="\xBD\uFFFD\u0000hR \uFFFD:`uN\uFFFD\uFFFDmessage\uFFFD" record=nil 2025-06-18 11:16:49 +0900 [warn]: #0 skip invalid event: host="127.0.0.1" tag="incoming" time=50 record=nil 2025-06-18 11:16:49 +0900 [warn]: #0 skip invalid event: host="127.0.0.1" tag="incoming" time=97 record=nil ... ``` And, when `require_ack_response` was configured to `true` in out_forward, 1. in_forward causes `error_class=TypeError error="can't convert String into an exact number"` error by broken data. 2. in_forward can't sent the ack response by above error. 3. out_forward causes `error_class=Fluent::Plugin::ForwardOutput::NoNodesAvailable error="no nodes are available"` error because in_forward doesn't send the ack response. 4. out_forward will retry to send broken data. 5. back to `1.` in_forward and out_forward will repeat errors and retries indefinitely. If `skip_invalid_event` was configured to `true` in in_forward, in_forward doesn't cause the error by broken data, in_forward and out_forward don't repeat errors and retries. So, Fluentd will be more stable by enabled `skip_invalid_event`. ### About performance degradation It shows the processing time of `in_forward` when receiving 1 GB of data. * with `skip_invalid_event false` * 15.178013304 sec * with `skip_invalid_event true` * 15.859435514 sec Even when skip_invalid_event was enabled, I think the performance degradation was minimal. **Docs Changes**: fluent/fluentd-docs-gitbook#587 **Release Note**: Same as the title. Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fluent/fluentd#5003