fix(kafka): add message processing failure counter#477
fix(kafka): add message processing failure counter#477dscervantes wants to merge 3 commits intoRedHatInsights:masterfrom
Conversation
RHCLOUD-45237 Wire up the existing IncMessageProcessErrors() prometheus counter at every failure return path in the kafka message handler: JSON unmarshal errors, payload upsert failures, status/service/source table entry creation errors, and DB insert retry exhaustion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RHCLOUD-45237 Also count invalid request ID as a message processing error for consistency with all other early-return error paths in onMessage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
dehort
left a comment
dehort
left a comment
There was a problem hiding this comment.
Left a couple of comments. Other than that, it looks good to me!
| } else { | ||
| l.Log.Error("ERROR: Unmarshaling Payload Status Event: ", err) | ||
| } | ||
| endpoints.IncMessageProcessErrors() |
There was a problem hiding this comment.
I think this should be IncInvalidConsumerRequestIDs().
There was a problem hiding this comment.
Good point — changed to IncInvalidConsumerRequestIDs(). Also moved the counter out of validateRequestID() so the validation function stays side-effect-free and the counter is only at the call site (avoids double-counting). Fixed in 08e148d.
| } | ||
|
|
||
| if !validateRequestID(cfg.RequestConfig.ValidateRequestIDLength, payloadStatus.RequestID) { | ||
| endpoints.IncMessageProcessErrors() |
There was a problem hiding this comment.
I think this should be IncInvalidConsumerRequestIDs().
There was a problem hiding this comment.
Done — changed to IncInvalidConsumerRequestIDs() here as well. Since validateRequestID() was calling it internally, I removed it from there to avoid double-counting on the same counter.
RHCLOUD-45237 Use IncInvalidConsumerRequestIDs() instead of IncMessageProcessErrors() for unmarshal and request ID validation failures. Move counter out of validateRequestID to the call site to avoid double-counting and keep the validation function free of side effects. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3 participants
Summary
payload_tracker_message_process_errorsPrometheus counter (IncMessageProcessErrors()) at every failure return path in the Kafka message handlerinternal/endpoints/metrics.gobut were never calledRHCLOUD-45237
Test plan
go build ./...passes (confirmed locally)go vet ./...shows no new warnings (confirmed — existing warnings are pre-existing)payload_tracker_message_process_errorsmetric increments on message processing failures in staging🤖 Generated with Claude Code