feat: implement ZSTD compression pooling and integrate across project

[kalbasit]

This change introduces a new pkg/compress package that provides a pool
of ZSTD encoders and decoders. This optimization reduces allocation
overhead during high-concurrency compression and decompression
operations.

The implementation:

• Uses sync.Pool for managing zstd.Encoder and zstd.Decoder instances.
• Provides PooledZstdWriter and PooledZstdReader wrappers for easy
  resource management.
• Integrates the pool into the local chunk store for chunk compression.
• Integrates the pool into the server middleware for HTTP response
  compression.
• Updates NAR reading and test utilities to use the centralized
  compression package.

This was needed to improve the efficiency of the cache server when
handling many small and large NAR files simultaneously, minimizing
garbage collection pressure.

[dosubot]

Related Documentation

Checked 7 published document(s) in 1 knowledge base(s). No updates required.

^{How did I do? Any feedback?}  

[kalbasit]

• test: improve and speed up testing #867
• feat: implement ZSTD compression pooling and integrate across project #865 👈 (View in Graphite)
• test: add end-to-end test for CDC chunk compression #864
• feat: add zstd compression to s3 chunk store #863
• feat: add zstd compression to local chunk store #862
• refactor: update PutChunk interface to return compressed_size #861
• feat: add compressed_size column to chunks table for CDC compression tracking #860
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[gemini-code-assist]

Summary of Changes

Hello @kalbasit, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a fundamental optimization for ZSTD compression and decompression operations across the project. By centralizing and pooling ZSTD encoder and decoder instances, the system can now handle high-throughput scenarios with reduced memory allocations and lower garbage collection overhead. This change aims to enhance the overall efficiency and scalability of the cache server, particularly when dealing with numerous NAR files concurrently.

Highlights

• New ZSTD Compression Pooling Package: Introduced a new pkg/compress package that provides a centralized, thread-safe sync.Pool-based mechanism for recycling ZSTD encoder and decoder instances. This significantly reduces allocation overhead and garbage collection pressure during high-concurrency compression/decompression operations.
• Integration Across Core Components: The newly implemented ZSTD pooling has been integrated into critical parts of the project, including the local chunk store for NAR compression, the server middleware for HTTP response compression, and various NAR reading and test utilities, ensuring consistent performance improvements.
• Improved Resource Management: The pooling mechanism offers both low-level (manual Get/Put) and high-level (wrapped PooledZstdWriter/PooledZstdReader with automatic Close handling) APIs, simplifying resource management and promoting cleaner code while ensuring instances are properly reset before reuse.
• Comprehensive Documentation and Migration Guide: Detailed documentation (README_ZSTD.md, ZSTD_POOL.md, ZSTD_QUICK_REF.md) and a migration guide (MIGRATION.md) have been added to facilitate understanding and adoption of the new compression pooling system, including examples and a migration checklist.

Changelog

• pkg/cache/cache_internal_test.go
  • Removed direct import of github.com/klauspost/compress/zstd.
  • Imported github.com/kalbasit/ncps/pkg/compress.
  • Updated ZSTD writer creation in testRunLRU to use compress.GetZstdWriter() and compress.PutZstdWriter().
• pkg/cache/cache_test.go
  • Removed direct import of github.com/klauspost/compress/zstd.
  • Imported github.com/kalbasit/ncps/pkg/compress.
  • Updated ZSTD reader creation in testGetNarInfo to use compress.GetZstdReader() and compress.PutZstdReader().
• pkg/cache/export_test.go
  • Removed direct import of github.com/klauspost/compress/zstd.
  • Imported github.com/kalbasit/ncps/pkg/compress.
  • Updated compressZstd function to use compress.NewPooledZstdWriter() for ZSTD compression.
• pkg/compress/MIGRATION.md
  • Added a new markdown file detailing migration examples for various packages to use the new ZSTD pool, including before/after code snippets and a migration checklist.
• pkg/compress/README_ZSTD.md
  • Added a new markdown file providing an overview and summary of the ZSTD pool implementation, its key features, usage examples, and integration points.
• pkg/compress/ZSTD_POOL.md
  • Added a new markdown file offering complete API documentation for the ZSTD pool, covering low-level and high-level APIs, usage examples, configuration, and best practices.
• pkg/compress/ZSTD_QUICK_REF.md
  • Added a new markdown file providing a quick reference guide for the ZSTD pool, including common patterns, an API cheat sheet, and best practices.
• pkg/compress/zstd.go
  • Added a new Go file implementing zstdWriterPool and zstdReaderPool using sync.Pool.
  • Provided GetZstdWriter(), PutZstdWriter(), GetZstdReader(), and PutZstdReader() functions for manual pool management.
  • Introduced PooledZstdWriter and PooledZstdReader structs with NewPooledZstdWriter() and NewPooledZstdReader() constructors for automatic pool management and resource cleanup.
• pkg/compress/zstd_test.go
  • Added a new Go test file with comprehensive tests for the ZSTD pooling functionality, covering get/put operations, pooled wrapper behavior, round-trip compression/decompression, error handling, and resource reuse.
• pkg/nar/reader.go
  • Removed direct import of github.com/klauspost/compress/zstd.
  • Imported github.com/kalbasit/ncps/pkg/compress.
  • Updated DecompressReader function to use compress.NewPooledZstdReader() for ZSTD decompression.
• pkg/nar/reader_test.go
  • Removed direct import of github.com/klauspost/compress/zstd.
  • Imported github.com/kalbasit/ncps/pkg/compress.
  • Updated ZSTD writer creation in TestDecompressReader to use compress.NewPooledZstdWriter().
• pkg/server/server.go
  • Removed sync and github.com/klauspost/compress/zstd imports.
  • Removed the local zstdWriterPool global variable.
  • Imported github.com/kalbasit/ncps/pkg/compress.
  • Updated getNar handler to use compress.NewPooledZstdWriter() for HTTP response compression.
• pkg/server/server_test.go
  • Removed direct import of github.com/klauspost/compress/zstd.
  • Imported github.com/kalbasit/ncps/pkg/compress.
  • Updated ZSTD reader creation in TestGetNar_ZstdCompression to use compress.NewPooledZstdReader().
• pkg/storage/chunk/local.go
  • Removed direct import of github.com/klauspost/compress/zstd.
  • Imported github.com/kalbasit/ncps/pkg/compress.
  • Refactored localReadCloser to embed *compress.PooledZstdReader instead of *zstd.Decoder.
  • Removed encoder and decoder fields from localStore struct.
  • Updated NewLocalStore to no longer initialize ZSTD encoder/decoder instances directly.
  • Modified GetChunk to use compress.NewPooledZstdReader() for decompression.
  • Modified PutChunk to use compress.GetZstdWriter() and compress.PutZstdWriter() for compression.
• pkg/storage/chunk/s3.go
  • Removed direct import of github.com/klauspost/compress/zstd.
  • Imported github.com/kalbasit/ncps/pkg/compress.
  • Refactored s3ReadCloser to embed *compress.PooledZstdReader instead of *zstd.Decoder.
  • Removed encoder and decoder fields from s3Store struct.
  • Updated NewS3Store to no longer initialize ZSTD encoder/decoder instances directly.
  • Modified GetChunk to use compress.NewPooledZstdReader() for decompression.
  • Modified PutChunk to use compress.GetZstdWriter() and compress.PutZstdWriter() for compression.
• testdata/server.go
  • Removed direct import of github.com/klauspost/compress/zstd.
  • Imported github.com/kalbasit/ncps/pkg/compress.
  • Updated compressMiddleware to use compress.NewPooledZstdWriter() for response compression.
• testdata/server_test.go
  • Removed direct import of github.com/klauspost/compress/zstd.
  • Imported github.com/kalbasit/ncps/pkg/compress.
  • Updated ZSTD reader creation in TestNewTestServerWithZSTD to use compress.GetZstdReader() and compress.PutZstdReader().

Activity

• No human activity has been recorded on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a centralized ZSTD compression pool to improve performance by reducing allocation overhead, which is a great optimization. The implementation is well-structured with both low-level and high-level APIs for managing pooled encoders and decoders. The new pkg/compress is cleanly integrated across various parts of the project, including storage backends, server middleware, and test utilities, replacing previous ad-hoc compression logic. My main feedback concerns the new documentation files, which consistently reference an incorrect package path (pkg/helper instead of pkg/compress). The provided comments highlight these issues and suggest corrections, which are all valid and align with the project's standards.

[kalbasit]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a new pkg/zstd package to pool ZSTD encoders and decoders, which is a great optimization for reducing allocation overhead in high-concurrency scenarios. The implementation is well-structured with wrappers for automatic resource management, and it has been integrated across various parts of the project, including chunk storage and server middleware. The changes are logical and improve performance. However, there is a widespread renaming of terms like 'compress' to 'zstded' and 'decompress' to 'dezstd', which makes the code awkward to read and less maintainable. I've added a few comments to highlight this issue.

I am having trouble creating individual review comments. Click here to see my feedback.

pkg/cache/cache.go (1349)

The variable compressedSize has been renamed to zstdedSize. While the intent is to be specific about the compression algorithm, 'zstded' is not a standard English word and can be confusing. It also makes the code less generic. Consider using zstdCompressedSize or reverting to compressedSize and relying on context to understand the algorithm. This comment applies to similar renamings throughout the PR (e.g., zstdible, dezstded).

_, compressedSize, err := chunkStore.PutChunk(ctx, chunkMetadata.Hash, chunkMetadata.Data)

pkg/cache/cdc_test.go (336)

The string 'compressible' has been changed to 'zstdible'. This is not a standard term and harms readability. Please consider using more conventional terms like 'compressible' or 'zstd_compressible'.

content := strings.Repeat("compressible", 1000)

pkg/zstd/README.md (24)

The example code uses helper.NewPooledWriter. It seems helper should be zstd, as the package being documented is zstd. This appears to be a copy-paste error that is repeated throughout the README. Please correct this to avoid confusion for developers using this new package.

pw := zstd.NewPooledWriter(&buf)

[kalbasit]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a well-designed zstd package for pooling ZSTD encoders and decoders, which is a great optimization for reducing allocations and GC pressure. The integration across the codebase is clean and effectively uses the new pooling mechanism. The test suite has also been improved, particularly with more robust race condition testing for S3 storage. My feedback is minor, focusing on improving the new documentation and refining error handling in one of the test helpers to align with best practices for test cleanup.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.96%. Comparing base (42ad9bb) to head (ff9c82d).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #865   +/-   ##
=====================================
  Coverage   3.96%   3.96%           
=====================================
  Files          6       6           
  Lines        429     429           
=====================================
  Hits          17      17           
  Misses       409     409           
  Partials       3       3           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.