feat(clickhouse): db cleanup #1876
Conversation
|
Really nice thanks! |
| metadata: | ||
| annotations: | ||
| checksum/configYml: {{ .Values.config.configYml | toYaml | toString | sha256sum }} | ||
| checksum/config.yaml: {{ include "snuba.config" . | sha256sum }} |
There was a problem hiding this comment.
@Mokto @TartanLeGrand This should be "sentry.snuba.config"
|
@TartanLeGrand why did we merge this. These tables are supposed to autocleanup by their configured retention policy. |
|
|
@TartanLeGrand I think I am missing something while looking at that referenced issue. From what I can see, Snuba is respecting our retention policy (oldest record I can see in CH is 90 days old). While this change is probably useful, I think enabling it by default with a shorter retention period than the built-in cleanup ( |
No, that is node store which is in postgres not in snuba. |
This pull request introduces a new automated cleanup process for ClickHouse tables in Snuba, configurable via Helm chart values. The main changes add a CronJob resource for periodic cleanup, enable flexible table selection (auto-detection or fallback list), and provide configuration options for scheduling, retention, and resource management. It also adds a dedicated ServiceAccount for the cleanup job.
ClickHouse Cleanup CronJob Implementation:
cronjob-clickhouse-cleanup.yamlthat creates aCronJobto periodically delete old data from ClickHouse tables based on a configurable retention period. The job supports auto-discovery of tables using include/exclude regex patterns, with a fallback hardcoded list if auto-detection fails.snuba.cleanupinvalues.yamlto control job scheduling, retention period, resource limits, table selection patterns, and other pod/job settings.ServiceAccount and Permissions:
serviceaccount-clickhouse-cleanup.yamlto create a dedicated ServiceAccount for the cleanup job, with customizable annotations and token settings.Cleanup and Refactoring:
symbolicator.cleanupsection fromvalues.yamlto avoid confusion and maintain chart clarity.