Adds option to delete extracted tag attributes #78
Conversation
|
/merge |
|
View all feedbacks in Devflow UI.
This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
This pull request was merged directly. |
|
/merge |
|
View all feedbacks in Devflow UI.
PR already in the queue with status waiting |
dd-devflow
bot
added
mergequeue-status: done
and removed
mergequeue-status: waiting
labels
2 participants
What does this PR do?
Currently this plugin duplicates docker and kubernetes attributes into tags. While useful, this means that the logs uploaded to datadog are now larger than they otherwise would be, leading to increased bytes ingested and therefore increased costs. This PR aims to provide a configuration option to delete kubernetes and docker attributes from the log after the relevant information has been extracted into tags.
This should address the issue #42 that has been open for 5 years at this point.
Motivation
Removing this duplicative data should reduce load and costs for people using the plugin. In a test from logs on my applications I was seeing a roughly 30% reduction in overall bytes ingested.
Additional Notes
This change intentionally removes all docker and kubernetes attributes from the logs after the relevant tags are extracted, not limited to just the ones that are specifically added as tags as part of the plugin. Many of the attributes are already included elsewhere as tags, and IMO the improvement from cleaning them all up is worth the trade off of potentially needing to map a couple custom kubernetes / docker attributes to a different spot in the record if someone does not want them removed.