feat: add zstd compression to local chunk store

[kalbasit]

Implement Zstandard (zstd) compression for the local chunk store to reduce disk usage for cached chunks.

Changes:

• Added zstd encoder to the localStore struct for efficient compression.
• Modified PutChunk to compress data using zstd before writing to disk and return the compressed size.
• Updated GetChunk to use zstd.NewReader for transparent decompression, wrapped in a localReadCloser to ensure proper resource cleanup.
• Added comprehensive tests to verify compression efficiency and data integrity through round-trip checks.

This change reduces the storage footprint of the local chunk cache, which is especially beneficial for large Nix store paths.

[kalbasit]

• test: improve and speed up testing #867
• feat: implement ZSTD compression pooling and integrate across project #865
• test: add end-to-end test for CDC chunk compression #864
• feat: add zstd compression to s3 chunk store #863
• feat: add zstd compression to local chunk store #862 👈 (View in Graphite)
• refactor: update PutChunk interface to return compressed_size #861
• feat: add compressed_size column to chunks table for CDC compression tracking #860
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[gemini-code-assist]

Summary of Changes

Hello @kalbasit, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces Zstandard compression to the local chunk storage mechanism. The primary goal is to significantly reduce the disk footprint of cached data, which is particularly beneficial for environments with large data volumes like Nix store paths. The changes ensure that data is compressed before being written and decompressed seamlessly when read, without altering the external API of the chunk store.

Highlights

• Zstandard Compression Integration: Integrated Zstandard (zstd) compression into the local chunk store to reduce disk usage for cached chunks.
• Compression on Write: Modified the PutChunk function to compress data using the zstd encoder before writing it to disk, and updated it to return the compressed size.
• Decompression on Read: Updated the GetChunk function to transparently decompress data using a zstd decoder, wrapping the file and decoder in a custom localReadCloser for proper resource cleanup.
• Comprehensive Testing: Added new test cases to verify the efficiency of zstd compression and ensure data integrity through round-trip compression and decompression checks.

Changelog

• pkg/storage/chunk/local.go
  • Imported the github.com/klauspost/compress/zstd package.
  • Defined a new localReadCloser struct to manage the lifecycle of zstd decoders and files.
  • Added encoder and decoder fields to the localStore struct.
  • Initialized zstd encoder and decoder instances in NewLocalStore, ensuring proper error handling and resource cleanup.
  • Modified GetChunk to wrap the opened file with a zstd decoder and return a localReadCloser.
  • Updated PutChunk to compress the chunk data using the zstd encoder before writing it to a temporary file.
  • Changed PutChunk's return value to include the actual compressed size of the data.
• pkg/storage/chunk/local_test.go
  • Imported the bytes package for test data manipulation.
  • Added a test case stored chunk is zstd-compressed on disk to assert that compressed data size is less than original for compressible data.
  • Added a test case compressed chunk round-trips correctly to verify that data can be compressed and then decompressed back to its original form.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[dosubot]

Related Documentation

Checked 7 published document(s) in 1 knowledge base(s). No updates required.

^{How did I do? Any feedback?}  

[gemini-code-assist]

Code Review

This pull request introduces zstd compression to the local chunk store, aiming to reduce disk usage by compressing data on PutChunk and decompressing it on GetChunk. However, a significant race condition exists in the PutChunk implementation due to the use of a shared, unsynchronized zstd.Encoder instance. Since klauspost/compress/zstd's zstd.Encoder is not thread-safe, concurrent calls to PutChunk could lead to data corruption or application crashes. It is recommended to implement a sync.Pool for encoders or use a mutex to serialize access. Additionally, there are a couple of suggestions for improvement: remove the unused decoder field in the localStore struct to simplify the code, and refactor localReadCloser to use composition instead of embedding for a more explicit and robust API.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.96%. Comparing base (152eeb6) to head (c933ddf).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #862   +/-   ##
=====================================
  Coverage   3.96%   3.96%           
=====================================
  Files          6       6           
  Lines        429     429           
=====================================
  Hits          17      17           
  Misses       409     409           
  Partials       3       3           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.