Skip to content

Update deps / python to latest supported version - use UV for dep management #405

@svij-sc

Description

@svij-sc

Python3.9 is reaching has reached end of life on October 31.
This poses some concerns incase there are any security issues w/ python or any of the deps we use as they will likely not publish any patches, etc.
So a need to update python version and deps exist - which is quite heavy handed change as we make use of libraries that are not all available on pypi.

  1. Thus, the dependency management is complicated and involves us indexing wheels directly:

    "torch @ https://download.pytorch.org/whl/cu121/torch-2.5.1%2Bcu121-cp39-cp39-linux_x86_64.whl#sha256=3c96b2ec4723e7d97259964ee73e2d6a2bace42511a49005b083ea7be1a0b0ac ; platform_system=='Linux' and python_version=='3.9'",
    # ===================================== Deps for PyTorch Geometric =====================================
    "pyg-lib @ https://data.pyg.org/whl/torch-2.5.0+cu121/pyg_lib-0.4.0+pt25cu121-cp39-cp39-linux_x86_64.whl ; platform_system=='Linux' and python_version=='3.9'",
    "torch-cluster @ https://data.pyg.org/whl/torch-2.5.0+cu121/torch_cluster-1.6.3+pt25cu121-cp39-cp39-linux_x86_64.whl ; platform_system=='Linux' and python_version=='3.9'",
    "torch-scatter @ https://data.pyg.org/whl/torch-2.5.0+cu121/torch_scatter-2.1.2+pt25cu121-cp39-cp39-linux_x86_64.whl ; platform_system=='Linux' and python_version=='3.9'",
    "torch-sparse @ https://data.pyg.org/whl/torch-2.5.0+cu121/torch_sparse-0.6.18+pt25cu121-cp39-cp39-linux_x86_64.whl ; platform_system=='Linux' and python_version=='3.9'",
    "torch-spline-conv @ https://data.pyg.org/whl/torch-2.5.0+cu121/torch_spline_conv-1.2.2+pt25cu121-cp39-cp39-linux_x86_64.whl ; platform_system=='Linux' and python_version=='3.9'",
    # ===================================== Deps for TorchRec =====================================
    "torchrec @ https://download.pytorch.org/whl/cu121/torchrec-1.0.0%2Bcu121-py3-none-any.whl ; platform_system=='Linux' and python_version=='3.9'",
    "fbgemm-gpu @ https://download.pytorch.org/whl/cu121/fbgemm_gpu-1.0.0%2Bcu121-cp39-cp39-manylinux2014_x86_64.whl ; platform_system=='Linux' and python_version=='3.9'",

  2. Secondly, to be most secure, this involves us maintaining seperate frozen requirements for each host we support:
    https://github.com/Snapchat/GiGL/tree/main/requirements

  3. Finally, there are some other complications of trying to package and host wheels in corp/private registries: https://go/gigl-issue/1434

Proposal to make this easier
Taking these issues into account, we can make our dependencies easier to manage if we adopt uv as part of upgrading our deps / python version: https://docs.astral.sh/uv/

Specifically, the way it helps us is:

  1. uv allows us to more easily maintain custom indexes for packages: https://docs.astral.sh/uv/concepts/indexes/
  2. uv maintains a lockfile for us that contains information for all hosts / all dependency group resolutions: https://docs.astral.sh/uv/guides/projects/#uvlock
  3. The customizability of custom auth paths/certs allows practitioners to more easy package host wheels in corp/private registries: https://docs.astral.sh/uv/concepts/authentication/

Beyond our issues above, uv further improves the developer experience by:

  1. Installing and managing our python environment for us
  2. It replaces our use of multiple other tools: pip, pip-tools, twine, conda, etc.
  3. Its support for workspace allows us to manage GiGL as part of other larger codebases.
  4. Container like sandboxing

Sub-issues

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions