Build Python wheels for git-annex and upload them to PyPi

[mih]

I propose to build Python wheels for git-annex...

• to be able to declare proper "native" dependencies on it
• to make it accessible to venv management (e.g. by uv) with download/installation caching
• to spare users the need to use the custom datalad-installer to get git-annex deploy on the more challenging platforms

PyPi has a 60MB package limit. git-annex binaries are ~30MB (compressed). This would fit.

I include a patch to the git-annex sources below that enabled running

hatch build -t wheel

in a git-annex source tree that has been built via

stack setup
stack build --only-dependencies --extra-include-dirs=$PWD --extra-lib-dirs=$PWD
stack install --no-haddock --local-bin-path bin

to produce something like git_annex-10.20250417.dev28+g7bf6e1df8a-py3-none-linux_x86_64.whl. The stack parts of the build could be integrated more. However, I am proposing this draft already now, because some issues are unresolved and additional ideas are welcome. Those are:

git-annex-shell, git-remote-annex, and git-remote-tor-annex are not installed. These are merely symlinks to the git-annex binary, but: A python wheel is a ZIP file, and it does not support symlinks.

Worse, if I do not give a damn and include copies instead of symlinks, the wheel size multiplies 🙄

I assume that git-annex looks at argv[0] to decide what it should do, so adding a wrapper script is not really an option. I further assume that the git-annex installer for windows just creates these copies on the target system.

The wheel specification has no concept of a post-install step.

What could be done?

Here is the patch:

diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000000..f6888ad849
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,31 @@
+[project]
+name = "git-annex"
+dynamic = ["version"]
+description = "Add your description here"
+readme = {file = "README", content-type = "text/markdown"}
+requires-python = ">=3.9"
+
+[build-system]
+requires = [
+  "hatchling",
+  "hatch-vcs",
+]
+build-backend = "hatchling.build"
+
+[tool.hatch.version]
+source = "vcs"
+
+[tool.hatch.build.targets.wheel.shared-data]
+"bin/git-annex" = "bin/git-annex"
+
+[tool.hatch.build.targets.wheel]
+# we need to ship at least one file
+only-include = ["python/py.typed"]
+
+[tool.hatch.build.targets.wheel.sources]
+# give the 'only-include' files a base directory in the wheel
+"python" = "git-annex"
+
+[tool.hatch.build.targets.wheel.hooks.custom]
+# custom build hook to set some metadata
+path = "python/build_hook_plugin.py"
diff --git a/python/build_hook_plugin.py b/python/build_hook_plugin.py
new file mode 100644
index 0000000000..02f556a4a1
--- /dev/null
+++ b/python/build_hook_plugin.py
@@ -0,0 +1,18 @@
+from sysconfig import get_platform
+from typing import Any
+
+from hatchling.builders.hooks.plugin.interface import BuildHookInterface
+
+
+class SpecialBuildHook(BuildHookInterface):
+
+    def initialize(
+        self,
+        version: str,  # noqa: ARG002
+        build_data: dict[str, Any],
+    ) -> None:
+        # we have platform-specific builds
+        build_data['pure_python'] = False
+        # set a tag that says: any python3 for the build platform
+        build_data['tag'] = \
+            f'py3-none-{get_platform().replace("-", "_").replace(".", "_")}'
diff --git a/python/py.typed b/python/py.typed
new file mode 100644
index 0000000000..e69de29bb2

It goes without saying that more metadata could and should be added. Importantly, this patch already does dynamic versioning using the git-annex repository (tags).

[mih]

I am currently leaning towards adding a dedicated git-annex-finalize-installation script. Users of the wheel would be instructed to run it manually, dependent applications could also take care of that.

[joeyh]

git-annex does look at argv[0]. It would be pretty easy to make "git-annex git-annex-shell" run like "git-annex-shell", "git-annex git-remote-annex" run like git-remote-annex, and then you could include wrapper scripts.

I don't know which of those commands would be useful in this context and which not, but it seems better to include all of them just in case.

I wonder about C library dependencies of git-annex. Building with stack like that avoids some that may be less usually installed, like libmagic, but still the user would need libgmp, which is not in the Debian base system. And these deps may change over time.

Perhaps you could include the git-annex-standalone tarball and unpack it somewhere in your finalize script. That would avoid the problem. But it is ~53mb currently and not infeasible it would reach 60 mb, indeed it is already over that size when git-annex is built with other build flags enabled (eg libmagic).

[mih]

The following patch to the patch above incorporates the actual build process into the packaging. This should work on systems that have the haskell stack, zlib1g-dev, libmagic-dev, and libncurses5-dev.

diff --git a/python/build_hook_plugin.py b/python/build_hook_plugin.py
index 02f556a4a1..fdaaded827 100644
--- a/python/build_hook_plugin.py
+++ b/python/build_hook_plugin.py
@@ -1,3 +1,4 @@
+from subprocess import run
 from sysconfig import get_platform
 from typing import Any
 
@@ -16,3 +17,17 @@ class SpecialBuildHook(BuildHookInterface):
         # set a tag that says: any python3 for the build platform
         build_data['tag'] = \
             f'py3-none-{get_platform().replace("-", "_").replace(".", "_")}'
+
+        build_cmds = [
+            ['stack', 'setup'],
+            [
+                'stack', 'build', '--only-dependencies',
+                f'--extra-include-dirs={self.root}',
+                f'--extra-lib-dirs={self.root}',
+            ],
+            ['stack', 'build', '--flag', 'git-annex:MagicMime'],
+            ['stack', 'install', '--no-haddock', '--local-bin-path', 'bin'],
+        ]
+
+        for cmd in build_cmds:
+            run(cmd, cwd=self.root, check=True)

[joeyh]

As far as the mechanics of including this patch in git-annex go, I'm ok with it but would want to clarify in the comment that python/py.typed is an intentionally empty file (or make the file a comment explaining its purpose).

[joeyh]

If you're building with libmagic, I think it will crash in some circumstances when the user's system does not have magic installed.

[yarikoptic]

I wonder about C library dependencies of git-annex. Building with stack like that avoids some that may be less usually installed, like libmagic, but still the user would need libgmp, which is not in the Debian base system. And these deps may change over time.

could/should we embed them in static build as I guess happening in @kyleam 's https://dl.kyleam.com/git-annex/ ?

[joeyh]

Recompressing git-annex-standalone-amd64.tar.gz with xz dropped it from 53mb to 30mb, so if python has an xz decompressor, space would not be a problem

[mih]

As far as the mechanics of including this patch in git-annex go, I'm ok with it but would want to clarify in the comment that python/py.typed is an intentionally empty file (or make the file a comment explaining its purpose).

Thanks. In the final packaging, there would be no need for the py.typed file. I only use it as a hack to bypass some hurdles. The actual python content shipped in the package would be the post-install finalyzer script (likely).

Recompressing git-annex-standalone-amd64.tar.gz with xz dropped it from 53mb to 30mb, so if python has an xz decompressor, space would not be a problem

Python has xz. But that doesn't help here (or I am misunderstanding). The wheel package format is a ZIP file. That is part of the spec. And I am not aware of means to replace the compression method used for the ZIP.

https://packaging.python.org/en/latest/specifications/binary-distribution-format/

[joeyh]

I was thinking you could ship the git-annex-standalone-amd64.tar.xz inside the ZIP file and decompress/install it in your git-annex-finalize-installation script.