-
Notifications
You must be signed in to change notification settings - Fork 12
Description
Python3.9 is reaching has reached end of life on October 31.
This poses some concerns incase there are any security issues w/ python or any of the deps we use as they will likely not publish any patches, etc.
So a need to update python version and deps exist - which is quite heavy handed change as we make use of libraries that are not all available on pypi.
-
Thus, the dependency management is complicated and involves us indexing wheels directly:
Lines 72 to 83 in a48144b
"torch @ https://download.pytorch.org/whl/cu121/torch-2.5.1%2Bcu121-cp39-cp39-linux_x86_64.whl#sha256=3c96b2ec4723e7d97259964ee73e2d6a2bace42511a49005b083ea7be1a0b0ac ; platform_system=='Linux' and python_version=='3.9'", # ===================================== Deps for PyTorch Geometric ===================================== "pyg-lib @ https://data.pyg.org/whl/torch-2.5.0+cu121/pyg_lib-0.4.0+pt25cu121-cp39-cp39-linux_x86_64.whl ; platform_system=='Linux' and python_version=='3.9'", "torch-cluster @ https://data.pyg.org/whl/torch-2.5.0+cu121/torch_cluster-1.6.3+pt25cu121-cp39-cp39-linux_x86_64.whl ; platform_system=='Linux' and python_version=='3.9'", "torch-scatter @ https://data.pyg.org/whl/torch-2.5.0+cu121/torch_scatter-2.1.2+pt25cu121-cp39-cp39-linux_x86_64.whl ; platform_system=='Linux' and python_version=='3.9'", "torch-sparse @ https://data.pyg.org/whl/torch-2.5.0+cu121/torch_sparse-0.6.18+pt25cu121-cp39-cp39-linux_x86_64.whl ; platform_system=='Linux' and python_version=='3.9'", "torch-spline-conv @ https://data.pyg.org/whl/torch-2.5.0+cu121/torch_spline_conv-1.2.2+pt25cu121-cp39-cp39-linux_x86_64.whl ; platform_system=='Linux' and python_version=='3.9'", # ===================================== Deps for TorchRec ===================================== "torchrec @ https://download.pytorch.org/whl/cu121/torchrec-1.0.0%2Bcu121-py3-none-any.whl ; platform_system=='Linux' and python_version=='3.9'", "fbgemm-gpu @ https://download.pytorch.org/whl/cu121/fbgemm_gpu-1.0.0%2Bcu121-cp39-cp39-manylinux2014_x86_64.whl ; platform_system=='Linux' and python_version=='3.9'", -
Secondly, to be most secure, this involves us maintaining seperate frozen requirements for each host we support:
https://github.com/Snapchat/GiGL/tree/main/requirements -
Finally, there are some other complications of trying to package and host wheels in corp/private registries: https://go/gigl-issue/1434
Proposal to make this easier
Taking these issues into account, we can make our dependencies easier to manage if we adopt uv as part of upgrading our deps / python version: https://docs.astral.sh/uv/
Specifically, the way it helps us is:
uvallows us to more easily maintain custom indexes for packages: https://docs.astral.sh/uv/concepts/indexes/uvmaintains a lockfile for us that contains information for all hosts / all dependency group resolutions: https://docs.astral.sh/uv/guides/projects/#uvlock- The customizability of custom auth paths/certs allows practitioners to more easy package host wheels in corp/private registries: https://docs.astral.sh/uv/concepts/authentication/
Beyond our issues above, uv further improves the developer experience by:
- Installing and managing our python environment for us
- It replaces our use of multiple other tools: pip, pip-tools, twine, conda, etc.
- Its support for workspace allows us to manage GiGL as part of other larger codebases.
- Container like sandboxing