Compress raw payload

[simark]

Just a random idea, but we could compress the raw payload stored in the dynamic data table, since that's the table that is going to grow a lot over time.

I checked for the length of the values in that column in my inventory:

100-200 chars:      284
300-400 chars:     3575
400-500 chars:   100767
500-600 chars:     1200
600-700 chars:       42
700-800 chars:        2
1100-1200 chars:    291
1200-1300 chars:     27
1500-1600 chars:     33
1600-1700 chars:      1

I searched a bit about compression of small strings. The main suggestion I found was to use an algorithm that allows passing an initial dictionary already "trained" on a data set representative of what you want to compress. You just need to make sure to pass that same dictionary to the decompressor. So if we would generate such a dictionary and embed it in the source, and it would never change.

A pre-trained dictionary is not as important on long inputs, since the algorithm "optimizes" its dictionary as it goes, so it amortizes the impact of the beginning of the input being compressed with a default (non-optimal for this input) dictionary. But for short strings like this, the impact of the non-optimal portion is greater.

In the Python standard library, there's zlib we could use, with the zdict parameter:

https://docs.python.org/3/library/zlib.html#zlib.compressobj

There's also the pyzstd library:

https://pyzstd.readthedocs.io/en/latest/#zstd-dict

I made this test for fun with pyzstd, see how much we would gain by using a pre-training dictionary:

import sqlite3
import pyzstd

s = sqlite3.connect("/home/smarchi/src/CanadianTracker/inventory.db")
rows = s.execute("SELECT raw_payload from products_dynamic").fetchall()
rows = [x[0].encode() for x in rows]

print("Building dictionary")
d = pyzstd.train_dict(rows, 110 * 1024)

compressor_without_d = pyzstd.ZstdCompressor()
compressor_with_d = pyzstd.ZstdCompressor(zstd_dict=d)


def compress_without_d(r):
    return compressor_without_d.compress(r, mode=pyzstd.ZstdCompressor.FLUSH_FRAME)


def compress_with_d(r):
    return compressor_with_d.compress(r, mode=pyzstd.ZstdCompressor.FLUSH_FRAME)


print("Compressing without dict")
rows_compressed_without_d = [compress_without_d(r) for r in rows]
print("Compressing with dict")
rows_compressed_with_d = [compress_with_d(r) for r in rows]

size_uncompressed = sum([len(x) for x in rows])
size_without_d = sum([len(x) for x in rows_compressed_without_d])
size_with_d = sum([len(x) for x in rows_compressed_with_d])

print(f"Size uncompressed: {size_uncompressed // 1024 // 1024} MiB")
print(f"Size without dict: {size_without_d // 1024 // 1024} MiB")
print(f"Size with dict: {size_with_d // 1024 // 1024} MiB")

Results are:

Size uncompressed: 46 MiB
Size without dict: 33 MiB
Size with dict: 7 MiB

So, it seems worth it to me.

[jgalar]

Interesting. That does look like something we should pursue.

My only gripe is the way the dictionary would be stored. I'm not sure I like the approach of storing it directly in the source. If we go in that direction I'd rather we store it as a proper zstd dictionnary file. It seems like a properly documented format (it's the output of $ zstd --train) and I guess pyzstd has a way to just dump a dict in that format.

For discussion's sake, I was wondering what a "clean" way of doing this would look like. I guess we could store the dict in a new table in the DB with a unique ID, and store the compressed fields along with this ID foreign key to know which dictionary was used.

I think it's what this project does, very roughly (using various dictids).
https://github.com/phiresky/sqlite-zstd

Anyway, it's most likely overkill for us, but fun to look into :)

What do you think of using a separate file?

[simark]

For my current implementation (not ready to be submitted), I generated a proper dictionary file. I initially wanted to distribute it as a separate data file in the Python package, but I did not quickly find how to do it, so I simply embedded it in a source file.
I don't think it's necessary to make the entries point to the dictionary used to compress them, because there will only be one well-known dictionary for the project. But if that ever changes and we have a reason to use more than one dictionary, then we still have an upgrade path, we can add a column to the table to add that reference. But until then, I'd keep things simple.