Don't know if this is outside the scope of this project, but it would be nice to have a PigzFile representing a file object like most stdlib implementations - gzip/bz2/lzma, etc. pyzstd and lz4 are also good examples.
More precisely a class with open(), read(), write() methods or a lower level (De)compressor objects that allow (de)compressing arbitrary chunks or stream of data.
A suggestion to how this could be implemented is to write to a buffer or file-like object in memory and once the block size is reached distribute the data across the workers, compress it and write it to the output file object.
Currently as far as I can see pigz-python only provides a single method for processing everything at once with internal r/w threads and doesn't allow specifying a file object or even custom path.
Don't know if this is outside the scope of this project, but it would be nice to have a PigzFile representing a file object like most stdlib implementations - gzip/bz2/lzma, etc. pyzstd and lz4 are also good examples.
More precisely a class with open(), read(), write() methods or a lower level (De)compressor objects that allow (de)compressing arbitrary chunks or stream of data.
A suggestion to how this could be implemented is to write to a buffer or file-like object in memory and once the block size is reached distribute the data across the workers, compress it and write it to the output file object.
Currently as far as I can see pigz-python only provides a single method for processing everything at once with internal r/w threads and doesn't allow specifying a file object or even custom path.