try sending uncompressed data

[cmatKhan]

is it faster?

[MackLiao]

Short answer: No.
However there are two other ways to make it faster after I deep-dived with Claude code:

• switch gzip from level 9 to level 1: it saves 92% of compression time, only 19% larger files
• Replace Python csv.writer with DuckDB native COPY TO, much faster serialization

which one would you like to implement?

[cmatKhan]

I tried the DuckDB copy to and didn't find it to be faster -- did you actually test it? If so and it did test faster, then yes. If no, then don't bother. My takeaway was that it was the compression that was taking the most time in the end. I didn't consider turning down the compression factor, though -- that is a good idea.

Do #1 for sure. Only do #2 if in actual testing it makes a difference