You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ic-python-db currently uses json.dumps/json.loads for all entity serialization in db_engine.py. While CPython 3.13 on WASI likely has the C-accelerated _json module, there is still overhead from:
JSON text format is larger than binary alternatives (string quoting, key repetition)
Every db.save() and db.load() goes through JSON encode/decode
On IC canisters, larger payloads mean more StableBTreeMap storage and more cycles
Proposal
Add msgpack as an alternative (or replacement) serialization format in db_engine.py.
Benefits
Smaller payloads — msgpack is typically 30-50% smaller than JSON for the same data
Faster encoding/decoding — even pure-Python msgpack is competitive with C-JSON for small dicts; a C-accelerated msgpack would be significantly faster
Binary-native — no string escaping overhead for keys/values
Implementation approach
The change is localized to db_engine.py — swap json.dumps/json.loads with msgpack equivalents:
# db_engine.pyimportmsgpack# or fallback to jsondefsave(self, type_name, id, data):
key=f"{type_name}@{id}"self._db_storage.insert(key, msgpack.packb(data))
defload(self, type_name, id):
key=f"{type_name}@{id}"data=self._db_storage.get(key)
ifdata:
returnmsgpack.unpackb(data)
returnNone
Considerations
Storage interface change: Storage.insert/get currently use str values. With msgpack the values would be bytes. Need to update Storage ABC and MemoryStorage.
Migration: Existing canister data is JSON-encoded. Need a migration path or dual-format support (try msgpack first, fall back to JSON).
WASI C extension: For maximum benefit, the msgpack C extension would need to be cross-compiled for wasm32-wasip1 and linked into the CPython WASM binary. Pure-Python msgpack may not outperform C-accelerated JSON.
Audit log: Also uses json.dumps — should be updated consistently.
Problem
ic-python-dbcurrently usesjson.dumps/json.loadsfor all entity serialization indb_engine.py. While CPython 3.13 on WASI likely has the C-accelerated_jsonmodule, there is still overhead from:db.save()anddb.load()goes through JSON encode/decodeProposal
Add msgpack as an alternative (or replacement) serialization format in
db_engine.py.Benefits
Implementation approach
The change is localized to
db_engine.py— swapjson.dumps/json.loadswith msgpack equivalents:Considerations
Storage.insert/getcurrently usestrvalues. With msgpack the values would bebytes. Need to updateStorageABC andMemoryStorage.wasm32-wasip1and linked into the CPython WASM binary. Pure-Python msgpack may not outperform C-accelerated JSON.json.dumps— should be updated consistently.