Skip to content

Comments

feat(engine): add query result retention policy (#123)#125

Merged
shefeek-jinnah merged 5 commits intomainfrom
shefeek/query_result_retention
Feb 25, 2026
Merged

feat(engine): add query result retention policy (#123)#125
shefeek-jinnah merged 5 commits intomainfrom
shefeek/query_result_retention

Conversation

@shefeek-jinnah
Copy link
Contributor

@shefeek-jinnah shefeek-jinnah commented Feb 19, 2026

Summary

  • Add background worker that periodically deletes expired query results (ready/failed) older than a configurable retention period
  • Schedule associated Parquet files for deletion when results are cleaned up
  • Configurable via result_retention_days (default: 7, set to 0 to disable)

Details

Complements the existing stale result cleanup worker (which marks stuck pending/processing results as failed) with a second phase that garbage-collects terminal results after a retention window.

  • Config: RUNTIMEDB_ENGINE__RESULT_RETENTION_DAYS env var or engine.result_retention_days in config
  • Worker: Runs every hour, deletes terminal results older than the cutoff and schedules their Parquet directories for deletion
  • Catalog: delete_expired_results implemented for Postgres (DELETE...RETURNING) and SQLite (SELECT + DELETE)
  • Lifecycle: Worker handle stored on RuntimeEngine, gracefully shut down during engine termination

@shefeek-jinnah shefeek-jinnah force-pushed the shefeek/query_result_retention branch from ae99390 to a7febc4 Compare February 25, 2026 05:23
@shefeek-jinnah shefeek-jinnah force-pushed the shefeek/query_result_retention branch from f7dfd5a to e657789 Compare February 25, 2026 10:10
@shefeek-jinnah shefeek-jinnah marked this pull request as ready for review February 25, 2026 12:02
@shefeek-jinnah shefeek-jinnah merged commit ee40bcd into main Feb 25, 2026
11 checks passed
@shefeek-jinnah shefeek-jinnah deleted the shefeek/query_result_retention branch February 25, 2026 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant