Add python wrapper

[TomBurdge]

Kudos!

Thanks for creating a cool rust interface with docker!

Pyo3 Implementation

I had some fun exploring this by creating a pyo3 implementation of the pull docker image command in my fork.

Releasing to PyPi

I would finish this with the other couple of commands you have completed, if you would be open to the contribution?

It would be fun to release this to pypi too.
I have some existing CI/CD for maturin releases.
The yaml might require some light changes to work with UV & docker.

Future feature: Python distribution

I know you mentioned distributing via apt-get/brew in the readme. It would be interesting to distribute the CLI tool itself via Python also. I'm sure it would be possible, the fun thing would be figuring out how to implement it in a DRY way.

[maxwellflitton]

Hey @TomBurdge thanks for checking it out. I'm going to work on pure C interfaces so we can install the lib in the root. We can then build Python, JavaScript, and other language interfaces. The approach could have the following C interface:

https://github.com/surrealdb/surrealml/blob/refactor/modules/c-wrapper/src/api/execution/buffered_compute.rs

and then it can be loaded via FFI in Python with the following approach:

https://github.com/surrealdb/surrealml/blob/refactor/modules/c-wrapper/tests/test_utils/c_lib_loader.py

Where we define C structs with the following:

https://github.com/surrealdb/surrealml/blob/refactor/modules/c-wrapper/tests/test_utils/return_structs.py

and we can interact with the raw C interface and python like the following:

https://github.com/surrealdb/surrealml/blob/e6ec80c34d4b1af708e82219ddd19e9c639d0da8/modules/c-wrapper/tests/api/execution/test_buffered_compute.py#L51

Because we know the raw C interface, this gives us the freedom to build clients in other languages.

[TomBurdge]

@maxwellflitton Thank you for the very complete explanation!

I have not yet had need to use the C interface directly.
I suppose it makes sense when you want to extend to multiple languages with one FFI - I will pay attention. :)

Python bin version of CLI

Since this is a CLI tool, would you consider (separately from the FFI) publishing the tool itself via pip?
This distributes dockpack, whereas the FFI works with coredockpack / core-dockpack.

I wanted to see how this could work, so I implemented this here.
This is very similar to the way that ruff does this, and works on my machine 😅 (Ubuntu).

Why distribute via pip?

1. Faster installation
   For me, installing dockpack via cargo takes a wee minute on my crusty laptop:

time cargo install dockpack
-- output:
-- real    0m10.600s
-- user    1m2.382s
-- sys     0m3.965s

This hasn't been distributed to pip, but for a library with near identical python source code:

time uv pip install ruff
-- output:
-- real    0m0.159s
-- user    0m0.023s
-- sys     0m0.024s

Probably a bit quicker with caching, but nontheless this is always going to be <= 1 second.

2. More accessible installation.

I have worked in very restrictive enterprise firewalls, but I haven't yet worked at a large organisation which blocks python and pip installations. It's just too important for the work (at least for someone who works in D&A).
Right now at [company redacted] I can't run rustup update, it's not allow listed on the outbound networking restrictions. But... I can run pip install for anything.

[maxwellflitton]

@TomBurdge thanks this is impressive. We are now precompiling dockpack and distributing them as assets, so it can just be downloaded by running the following script:

https://github.com/maxwellflitton/dockpack/blob/main/scripts/install_dockpack.sh

We then plan on building coredockpack and putting it in the root dir for all clients to access via FFI. This means all client packages are just their own native language loading the lib. I guess in the setup.py we can check to see if the coredockpack is there and we can download it if not on the pip install?

[TomBurdge]

Thanks @maxwellflitton that definitely makes sense.

I missed the script - we should probably add to the README that one can install dockpack with curl.

The bits I put in the pyproject.toml in the branch I linked also mean that this will be in the bin (with a thin python wrapper via the entry point script).

What I was thinking was two pip libraries:

• pydockpack: library with FFI entry points to call dockpack from within python code. (coredockpack crate).
• pydockpackcli: a way of distributing dockpack cli via pip (dockpack crate).

One last advantage: with uvx one would be able to run dockpack cli without even installing it - astral maintains pretty slim images with UV so users could make super quick automated runners for pulling images without even needing a cache.

Let me know what you think.

If you think it's easy enough to install via curl and bash then totally fair enough. I suppose we also need a good solution for Windows.

Other ways of distribution are going to be such a pain (apt-get looks confusing) that having one basically for free (I have plenty of boilerplate for maturin & pip ci/cd) it could be a good win.

[maxwellflitton]

Hey @TomBurdge I haven't forgotten this. It's just that I'm having a baby with my wife so been caught up with things but will get back to this when things settle down

[TomBurdge]

@maxwellflitton no worries at all. Best wishes to and your family. :)

[maxwellflitton]

@TomBurdge In terms of packaging, after the talk on dynamic C libs, do you think the move to dynamic C libs is a good shout? Below is the link to surrealML for the dynamic C lib wrapper:

https://github.com/surrealdb/surrealml/tree/tokenizers/modules/c-wrapper

Then we have a download script to the bin directory and then we can have different clients. Below is a link to the JavaScript and Python client as seen below:

https://github.com/surrealdb/surrealml/tree/tokenizers/clients

[TomBurdge]

@maxwellflitton added the c wrapper in #15 .

I did read the code SurrealML code. #15 has the core thrust of your priority: use the rust C binding rather than PyO3 macro magic that is: 1 too specific to python & makes it hard to also support other languages, 2 easy to get going but hard to debug when the macros topple.

I think using maturin (but not pyo3) will make distributing the code to PyPI realistic, which I think should be the minimum aim of an open source project to support python.