Implement LZ4 Compression Backend

[toymak3r]

Overview

Add LZ4 compression backend to the Goethe dialog system to provide an additional high-speed compression option.

Background

Currently, Goethe supports Zstd and Null compression backends. LZ4 offers excellent compression speed with reasonable compression ratios, making it ideal for scenarios where speed is prioritized over maximum compression.

Requirements

Core Implementation

• Implement LZ4CompressionBackend class in src/engine/core/compression/implementations/lz4.cpp
• Add corresponding header in include/goethe/lz4.hpp
• Register LZ4 backend in src/engine/core/compression/register_backends.cpp
• Update factory to include LZ4 in backend selection

Features

• Support for multiple compression levels (1-9)
• Configurable acceleration parameter
• Memory-efficient streaming compression
• Error handling and validation

Testing

• Unit tests for LZ4 backend in src/tests/test_compression.cpp
• Performance benchmarks vs Zstd and Null backends
• Memory usage tests
• Error condition tests

Integration

• Update CompressionManager to support LZ4
• Add LZ4 to automatic backend selection logic
• Update statistics tracking for LZ4 metrics
• Update command-line tools to support LZ4

Documentation

• Update API documentation
• Add LZ4 usage examples
• Update performance comparison documentation
• Add LZ4 configuration guide

Technical Details

Dependencies

• LZ4 library (system package or bundled)
• Update CMakeLists.txt to detect/find LZ4

Performance Targets

• Compression speed: >500 MB/s
• Decompression speed: >1000 MB/s
• Compression ratio: 2:1 to 3:1 (depending on data)

API Design

class LZ4CompressionBackend : public CompressionBackend {
public:
    LZ4CompressionBackend();
    ~LZ4CompressionBackend() override;
    
    void set_compression_level(int level) override;
    void set_acceleration(int acceleration);
    
    std::vector<uint8_t> compress(const std::vector<uint8_t>& data) override;
    std::vector<uint8_t> decompress(const std::vector<uint8_t>& data) override;
    
private:
    int compression_level_;
    int acceleration_;
};

Acceptance Criteria

• LZ4 backend compiles and links successfully
• All tests pass
• Performance benchmarks show LZ4 is faster than Zstd
• Integration tests verify LZ4 works with existing systems
• Documentation is complete and accurate

Priority

High - This is part of the short-term roadmap and provides immediate value to users who need fast compression.

Labels

• enhancement
• compression
• performance
• short-term
• high-priority

Estimated Effort

• Development: 2-3 weeks
• Testing: 1 week
• Documentation: 3-5 days