Collaboration proposal: Rust-powered QVD engine (qvdrs) — up to 350x faster

[bintocher]

Hi Constantin,

First of all — great work on PyQvd! It's the most well-known Python library for QVD files, and the API design is clean and well-documented.

I'm the author of qvdrs — a QVD library with the core engine written in Rust and Python bindings via PyO3 + Arrow zero-copy bridge (pip install qvdrs). I'd like to explore the possibility of collaboration.

Performance comparison

Benchmarks on the same machine with real QVD files, PyQvd 2.3.2 vs qvdrs 0.5.0:

Read

File	Rows	Cols	PyQvd	qvdrs	Speedup
11 KB	12	4	0.013s	0.000s	29x
62 KB	125	45	0.012s	0.001s	11x
2.3 MB	21,523	10	0.214s	0.011s	20x
35 MB	1,695,048	7	5.96s	0.26s	23x
480 MB	11,994,296	4	64.5s	2.1s	31x
560 MB	5,458,618	24	65.2s	3.9s	17x
1.7 GB	87,617,047	8	>10 min (killed)	23.4s	>25x

Write

File	Rows	Cols	PyQvd	qvdrs	Speedup
35 MB	1,695,048	7	7.8s	0.022s	351x
480 MB	11,994,296	4	50.9s	0.61s	83x

Features only in qvdrs

Feature	qvdrs
Streaming EXISTS() filtered read: 1.7GB, 87.6M rows -> 20.4M rows x 3 cols	9.0s
EXISTS() + save to QVD	13.3s
Parquet <-> QVD conversion	yes
DuckDB native integration (register QVD as SQL tables)	yes
DataFusion SQL queries on QVD	yes
Arrow RecordBatch zero-copy (pandas, Polars, DuckDB)	yes
CLI tool (convert, inspect, head, filter)	yes
Binary-identical output to Qlik Sense (MD5 verified)	yes

What each project brings

PyQvd — mature, clean API with QvdTable (filter_by, join, sort, append, insert), 25 stars, established user base, good documentation on readthedocs, pure Python — easy to understand and debug.

qvdrs — Rust core (17-350x faster), handles multi-GB files, streaming reader, EXISTS() filter (2.5x faster than Qlik Sense), Parquet/Arrow/DuckDB/DataFusion integration, binary-identical QVD output to Qlik Sense.

Development approach

The qvdrs codebase is developed with the help of Claude (Opus) — Anthropic's AI coding assistant. This significantly accelerates development: implementing new features, writing tests, debugging, and maintaining code quality. If you're open to collaboration, this is a powerful tool that can be used for joint development as well — it handles Rust, Python, and the QVD binary format equally well.

Proposal

A few possible directions:

1. Contribute to qvdrs — your QVD format expertise and API design skills would be very valuable. We could adopt PyQvd's richer QvdTable API (filter_by, join, sort, etc.) for the Python bindings.
2. qvdrs as an optional backend for PyQvd — keep PyQvd's API, but optionally use qvdrs for I/O (similar to how pandas uses pyarrow). Users get the familiar API with Rust performance.
3. Joint development — combine efforts under one project.

No pressure at all — just reaching out since we're both solving the same problem. Happy to discuss.

Stanislav
https://github.com/bintocher/qvdrs

[MuellerConstantin]

Hi Stanislav,

great project, thanks for reaching out. The performance differences are impressive; you can't achieve that with pure Python.

Your library definitely has better performance. I originally developed the library in pure Python for platform independence reasons; it was needed in my company. However, the real bottleneck in my implementation isn't necessarily Python itself, but primarily object creation and hash comparisons. I've also been working on the library's performance recently, but some design decisions in major version 2 have unfortunately limited my options.

Development approach

Interesting approach; how did you manage to get Claude to implement an undocumented proprietary format seemingly flawlessly?

Your Proposal

Sure, let's work together. We're both solving more or less the same problem. I especially like your second suggestion, possibly combined with your third.

I'm currently trying to get official/internal documentation of the QVD format from our Qlik partner; all the knowledge I've implemented and documented so far has only been gained through reverse engineering😆.

I can imagine rethinking the library's API in a major version 3 using the internal documentation and our current knowledge, writing the logic for platform independence primarily in Python, while simultaneously offering a Rust backend for speed. One could either create a monorepo with two packages or leave it at two repositories.

[project.optional-dependencies]
rust = ["qvdrs>=x.y.z"]

try:
    # rust implementation if installed
    from qvdrs import fast_reader
except ImportError:
    # fallback and platform independent python implementation
    from ._python_reader import fast_reader

pip install PyQvd
# or
pip install PyQvd[rust]

The result would be a Python library with a clean API that can be used platform-independently but also has a high-performance backend. And of course, a Rust library with minimal Python binding would also be included.

However, this would certainly require rethinking or adapting the APIs of both libraries. One could transition the collaborative work to a new major version.

---

What do you think of this type of collaboration?

Greetings
Constantin

[bintocher]

Hi Constantin,

Sorry for the late reply — was heads-down shipping new features.

Since the last conversation, qvdrs has moved pretty far ahead. Just in the latest release (v0.6.2):

• concatenate() — pure append with Qlik CONCATENATE semantics
• concatenate_with_pk() — PK-based upsert/dedup (replace/skip/error) — first QVD library in any language with this
• write_arrow() — PyArrow RecordBatch/Table → QVD directly, no Parquet roundtrip
• SchemaMode — strict/union validation for safe merges
• abi3 wheels for Python 3.9–3.14+

This is on top of what was already there: streaming reader, EXISTS() indexes, DuckDB/DataFusion SQL integration, Parquet conversion, pandas/Polars support, CLI tool, and byte-identical Qlik Sense output verified on 399 real files.

Regarding the official QVD documentation from your Qlik partner — that would certainly be interesting to validate against. Though at this point I suspect qvdrs already covers the full format spec. I've tested against hundreds of real QVD files across all symbol types (Int, Double, Text, DualInt, DualDouble), all edge cases (NULL sentinels, bit-stuffed indexes, bias values, varying BitWidth), and achieved byte-identical roundtrips with Qlik Sense output. The format turned out to be elegant but fully reverse-engineerable from the XML header + binary layout.

On performance — Qlik's QIX engine is written in C++, and their "optimized load" is supposed to be the fastest path for QVD files. But in my benchmarks, qvdrs wrapped in Python is still faster than Qlik Sense itself — both standard and optimized loads. You can easily verify this yourself — just compare qvd.read_qvd() timing against the same file loaded in Qlik. I'd be curious to see your numbers too.

I've been thinking about the best way to collaborate. Honestly, maintaining two separate implementations of the same format doesn't make much sense long-term — it's double the effort and double the bugs. qvdrs already covers everything PyQvd does and quite a bit more, with Rust performance underneath.

My proposal: join forces on qvdrs directly. You clearly understand the QVD format deeply and have great ideas about API design. That expertise would be incredibly valuable here. You could contribute to the Python API layer, help shape the interface, add features you care about — with full credit as a core contributor. And PyQvd could point users to qvdrs as the recommended library going forward (or become a thin wrapper around it).

The project is MIT-licensed, contributions are welcome, and I'd be happy to give you maintainer access.

Regarding Claude — it helped find publicly available QVD format descriptions online (community posts, Qlik help pages). The binary format details (symbol table encoding, bit-stuffed indexes, NULL sentinels, dual types) were figured out by testing against hundreds of real files until I got byte-identical output. Quite a puzzle, but a fun one.

What do you think?

Best,
Stanislav

[MuellerConstantin]

Hi Stanislav,

sorry for my late response. I've been thinking about this honestly. You're right that maintaining two separate implementations of the same format long-term doesn't make much sense. qvdrs is faster, has more features, and covers everything PyQvd does. I don't think there's a realistic scenario where I catch up on performance with a pure-Python implementation, nor would it make sense to try.
Here's what I'd like to do:

• I'll add a note to PyQvd's README and docs recommending qvdrs for anyone starting new projects or working with larger files.
• I'll keep PyQvd in maintenance mode (bugfixes, no new features) for existing users who depend on a zero-dependency pure-Python solution.

I'm planning to dig into the qvdrs codebase and would be happy to contribute directly, especially around API design and the QVD format documentation I'm trying to get from our Qlik partner. I'll start with some PRs once I've familiarized myself with the Rust/PyO3 architecture you chose.

Are you fine with that?

Best,
Constantin

[bintocher]

Hi Constantin,

Thanks for the thoughtful response — and honestly, this is exactly the outcome I was hoping for. Your willingness to step back from "my project vs. yours" and focus on the best long-term outcome for QVD users in the Python ecosystem says a lot. That kind of thinking is rare, and it's the reason I think this collaboration is going to work.

Your plan sounds great:

• Pointing PyQvd users toward qvdrs in the README is a perfect handoff for new projects.
• Keeping PyQvd in maintenance mode for the pure-Python use case makes total sense — there are real environments (restricted infrastructure, exotic platforms) where a zero-dependency Python implementation is the only option, and your library serves that niche well.
• Direct contributions to qvdrs — yes, absolutely. Looking forward to it.

I just sent you a collaborator invitation to bintocher/qvdrs with push access. That gives you the full contributor flow: branches, PRs, merging, issues. Welcome aboard.

A few areas where your expertise would be especially valuable, whenever you're ready:

1. API design as we move toward 1.0. qvdrs is approaching a stable 1.0 and the Python API is the right thing to lock down carefully. You've thought deeply about ergonomics from the PyQvd side — your perspective on naming, defaults, and the read/write surface would directly shape what the community uses for years.
2. Format documentation. If/when you get the official spec from your Qlik partner, that would be huge — both for validating qvdrs against edge cases we haven't seen and for producing the first publicly available authoritative QVD format reference. I'd be glad to co-author a FORMAT.md in the repo with you, crediting both reverse-engineering efforts and the official source.
3. Test coverage from real-world QVDs. You've likely accumulated test files and edge cases over the years that I haven't encountered. Anything you can contribute to the test suite makes the library more robust for everyone.
4. PyQvd → qvdrs migration guide. A short doc helping existing PyQvd users transition smoothly — written by you — would carry a lot more weight than anything I could write.

No pressure on timeline — start wherever feels natural after exploring the codebase. The repo is a single Rust crate (src/) with feature-gated bindings: src/python.rs for the PyO3 layer, src/napi.rs for the Node.js binding, src/parquet.rs and src/datafusion.rs for the data-stack integrations, and src/reader.rs / src/writer.rs / src/header.rs / src/symbol.rs for the core format logic. Docs are in docs/. Happy to walk you through any part — async or over a call, whichever you prefer.

Really glad we got here. Looking forward to building this together.

Best,
Stanislav

P.S. — once you're onboarded, I'll share the real story behind how qvdrs got reverse-engineered. There's a trick to it that I haven't put in writing anywhere yet — let's just say it goes beyond "test against real files until bytes match." Saving it for an in-person (well, async) handoff. 😉