feat: Python UV workflow #756

sriram-mv · 2025-06-20T18:59:22Z

Issue link: #743

Core Features:

Intelligent handling of pyproject.toml with automatic uv.lock detection
Multi-format support: pyproject.toml, requirements.txt, and requirements-*.txt variants
Lock file optimization: Automatic uv.lock usage for reproducible builds when available
Platform targeting: Lambda-compatible builds for x86_64 and ARM64 architectures
Python version control: Precise Python version targeting via UV's --python flags

UV Functionality:

uv sync --python X.Y for lock-based builds (pyproject.toml + uv.lock)
uv lock && uv export workflow for pyproject.toml without lock files
uv pip install --python-version X.Y --python-platform for requirements.txt
Proper virtual environment handling and site-packages extraction

Documentation:

Complete DESIGN.md with architecture overview and usage patterns
Accurate API documentation reflecting constructor + method call pattern
Smart dispatch logic clearly explained with examples
Updated to match actual implementation (no outdated references)

Compatibility:

Compatible with existing Lambda Builders interface
Supports standard build parameters (source_dir, artifacts_dir, scratch_dir)
Runtime parameter properly extracted and used for Python version targeting
Architecture parameter mapped to UV platform specifications

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

aws_lambda_builders/workflows/python_uv/utils.py

aws_lambda_builders/workflows/python_uv/validator.py

.github/workflows/build.yml

Vandita2020 · 2025-06-24T20:52:41Z

aws_lambda_builders/workflows/python_uv/exceptions.py

+class MissingUvError(LambdaBuilderError):
+    """Exception raised when UV executable is not found."""
+
+    MESSAGE = "uv executable not found in your environment. Please install uv: https://docs.astral.sh/uv/getting-started/installation/"


Can we also print the {uv_path} in the message, to mention the specific location where the uv executable was not found.

Based on how it's used here, I don't think we'd get an exact location back.

Vandita2020 · 2025-06-24T21:13:54Z

aws_lambda_builders/workflows/python_uv/packager.py

+        artifacts_dir_path: str,
+        scratch_dir_path: str,
+        manifest_path: str,
+        architecture: str = X86_64,


Why did we hardcode the architecture here?

Vandita2020 · 2025-06-24T21:21:16Z

aws_lambda_builders/workflows/python_uv/packager.py

+            handler = self._get_manifest_handler(manifest_name)
+
+            # Execute the handler
+            handler(manifest_path, artifacts_dir_path, scratch_dir_path, python_version, architecture, config)


Where is this method actually building the dependencies? For reference here is how python pip handles building the dependencies.

@sriram-mv can correct me if I'm wrong, but it seems like handler will be the function that is returned by _get_manifest_handler, which returns either _handle_pyproject_build or _build_from_requirements. Those go on to call one of the functions in UvRunner. I'm also writing this comment to confirm my own understanding.

Vandita2020 · 2025-06-24T21:31:11Z

aws_lambda_builders/workflows/python_uv/packager.py

+        except Exception as e:
+            raise UvBuildError(reason=f"Failed to build from pyproject.toml: {str(e)}")
+
+    def _export_pyproject_to_requirements(


Why are we converting pyproject.toml to requirements.txt, when it is recommended t o use pyproject.toml to define dependencies for UV?

This is something I also don't quite follow. It seems like if there's a lock file we use uv sync, and if there's not we convert to requirements and then use uv pip install. My understanding is that we can use uv sync even without the lock file, so I feel like the distinction between lock file and no lock file might not be useful. I could easily be overlooking something entirely, though.

• uv sync does NOT support --python-platform
• uv pip install DOES support --python-platform for cross-platform builds

this is the reason I believe.

Vandita2020 · 2025-06-24T21:34:03Z

aws_lambda_builders/workflows/python_uv/exceptions.py

+    MESSAGE = "Lock file operation failed: {reason}"
+
+
+class ManifestNotFoundError(LambdaBuilderError):


Where have we used this exception?

Vandita2020 · 2025-06-24T21:34:28Z

aws_lambda_builders/workflows/python_uv/exceptions.py

+    MESSAGE = "Failed to install dependencies using uv: {reason}"
+
+
+class UvResolutionError(LambdaBuilderError):


Where have we used this exception?

Side note to this, after changes are made for PR feedback, we need to make sure that DESIGN.md is updated accordingly as it is referenced there as well.

Vandita2020 · 2025-06-24T22:50:51Z

tests/unit/workflows/python_uv/test_packager.py

+    @patch("aws_lambda_builders.workflows.python_uv.packager.OSUtils")
+    def test_subprocess_uv_init_missing_uv(self, mock_osutils_class):
+        mock_osutils = Mock()
+        mock_osutils.which.return_value = None


Can we just keep the path here /usr/bin/

pkarakal · 2025-09-05T17:55:00Z

@sriram-mv do you think this could be further extended to also support poetry? Basically if an interface is included here (something like PythonPyprojectDepedenciesBuilder), then there can be a uv and a poetry adapters that implement the interface and just override some of the methods based on the cli arguments each tool has. It would also help address #688 which is a superset of #743.

AdrianCarreno · 2025-09-15T00:42:10Z

@sriram-mv @Vandita2020 will this PR ever be merged? UV has gained a lot of attention in the past year, and just using pip has been outdated for a long time in favor of faster and more reliable tools like poetry and UV.

reedham-aws · 2025-09-16T21:06:30Z

If there's not any movement on this PR in a couple weeks and I have some free time, I will try to update this PR with the feedback after taking a look.

sriram-mv · 2025-09-21T21:48:54Z

@reedham-aws I'll pick it up maybe later next week. Is there any other feedback that I can address in one go apart from what's mentioned?

reedham-aws

I didn't look too hard at the tests, and overall the code looks good. I have a couple code quality suggestions and a question about the way the code chooses to do uv pip install versus uv sync.

reedham-aws · 2025-09-24T22:01:41Z

aws_lambda_builders/workflows/python_uv/actions.py

+            # Determine target directory
+            target_artifact_dir = self.artifacts_dir
+            if self.dependencies_dir:
+                target_artifact_dir = self.dependencies_dir
+
+            # Build dependencies
+            package_builder.build_dependencies(
+                artifacts_dir_path=target_artifact_dir,
+                scratch_dir_path=self.scratch_dir,
+                manifest_path=self.manifest_path,
+                architecture=self.architecture,
+                config=self.config,
+            )


Can we put this section inside of a try/except like

aws-lambda-builders/aws_lambda_builders/workflows/python_pip/actions.py

Lines 68 to 80 in ee2a60b

try:

target_artifact_dir = self.artifacts_dir

# if dependencies folder is provided, download the dependencies into dependencies folder

if self.dependencies_dir:

target_artifact_dir = self.dependencies_dir

package_builder.build_dependencies(

artifacts_dir_path=target_artifact_dir,

scratch_dir_path=self.scratch_dir,

requirements_path=self.manifest_path,

)

except PackagerError as ex:

raise ActionFailedError(str(ex))

reedham-aws · 2025-09-24T22:12:51Z

aws_lambda_builders/workflows/python_uv/exceptions.py

+class MissingUvError(LambdaBuilderError):
+    """Exception raised when UV executable is not found."""
+
+    MESSAGE = "uv executable not found in your environment. Please install uv: https://docs.astral.sh/uv/getting-started/installation/"


Based on how it's used here, I don't think we'd get an exact location back.

reedham-aws · 2025-09-24T22:14:19Z

aws_lambda_builders/workflows/python_uv/exceptions.py

+    MESSAGE = "Failed to install dependencies using uv: {reason}"
+
+
+class UvResolutionError(LambdaBuilderError):


Side note to this, after changes are made for PR feedback, we need to make sure that DESIGN.md is updated accordingly as it is referenced there as well.

reedham-aws · 2025-09-24T22:18:14Z

aws_lambda_builders/workflows/python_uv/packager.py

+        except Exception as e:
+            raise UvBuildError(reason=f"Failed to build from requirements: {str(e)}")
+
+    def _extract_python_version(self, runtime: str) -> str:


Suggested change

def _extract_python_version(self, runtime: str) -> str:

def _extract_python_version(self, runtime: Optional[str]) -> str:

because when this is used the runtime is optional.

reedham-aws · 2025-09-24T22:20:26Z

aws_lambda_builders/workflows/python_uv/actions.py

+            raise ActionFailedError(f"UV build failed: {str(ex)}")
+
+
+class CopyDependenciesAction(BaseAction):


What is the reason to have this when a different CopyDependenciesAction exists?

reedham-aws · 2025-09-24T23:14:49Z

aws_lambda_builders/workflows/python_uv/packager.py

+        if not config.cache_dir:
+            uv_cache_dir = os.path.join(scratch_dir_path, "uv-cache")
+            # Use exist_ok equivalent for osutils
+            if not os.path.exists(uv_cache_dir):
+                self._osutils.makedirs(uv_cache_dir)
+            config.cache_dir = uv_cache_dir


nit: This same cache_dir setup is done in both of the main UvRunner functions, is probably redundant either here or there.

reedham-aws · 2025-09-24T23:18:01Z

aws_lambda_builders/workflows/python_uv/packager.py

+            handler = self._get_manifest_handler(manifest_name)
+
+            # Execute the handler
+            handler(manifest_path, artifacts_dir_path, scratch_dir_path, python_version, architecture, config)


@sriram-mv can correct me if I'm wrong, but it seems like handler will be the function that is returned by _get_manifest_handler, which returns either _handle_pyproject_build or _build_from_requirements. Those go on to call one of the functions in UvRunner. I'm also writing this comment to confirm my own understanding.

reedham-aws · 2025-09-24T23:19:54Z

aws_lambda_builders/workflows/python_uv/packager.py

+        """Get the appropriate handler function for a manifest file."""
+        # Exact match handlers for ACTUAL manifests
+        exact_handlers = {
+            "pyproject.toml": self._handle_pyproject_build,


Don't really have a problem with this structure, just curious if this is left as a dict so we can have further handlers in the future? Not understanding why this was done.

reedham-aws · 2025-09-24T23:25:31Z

aws_lambda_builders/workflows/python_uv/packager.py

+        except Exception as e:
+            raise UvBuildError(reason=f"Failed to build from pyproject.toml: {str(e)}")
+
+    def _export_pyproject_to_requirements(


This is something I also don't quite follow. It seems like if there's a lock file we use uv sync, and if there's not we convert to requirements and then use uv pip install. My understanding is that we can use uv sync even without the lock file, so I feel like the distinction between lock file and no lock file might not be useful. I could easily be overlooking something entirely, though.

reedham-aws · 2025-09-24T23:54:18Z

aws_lambda_builders/workflows/python_uv/workflow.py

+            )
+        ]
+
+    def _get_additional_binaries(self):


Is this needed for UV?

Core Features: * Intelligent handling of pyproject.toml with automatic uv.lock detection * Multi-format support: pyproject.toml, requirements.txt, and requirements-*.txt variants * Lock file optimization: Automatic uv.lock usage for reproducible builds when available * Platform targeting: Lambda-compatible builds for x86_64 and ARM64 architectures * Python version control: Precise Python version targeting via UV's --python flags UV Functionality: * `uv sync --python X.Y` for lock-based builds (pyproject.toml + uv.lock) * `uv lock && uv export` workflow for pyproject.toml without lock files * `uv pip install --python-version X.Y --python-platform` for requirements.txt * Proper virtual environment handling and site-packages extraction Documentation: * Complete DESIGN.md with architecture overview and usage patterns * Accurate API documentation reflecting constructor + method call pattern * Smart dispatch logic clearly explained with examples * Updated to match actual implementation (no outdated references) Compatibility: * Compatible with existing Lambda Builders interface * Supports standard build parameters (source_dir, artifacts_dir, scratch_dir) * Runtime parameter properly extracted and used for Python version targeting * Architecture parameter mapped to UV platform specifications

github-actions bot added the pr/internal label Jun 20, 2025

sriram-mv commented Jun 20, 2025

View reviewed changes

aws_lambda_builders/workflows/python_uv/utils.py Show resolved Hide resolved

aws_lambda_builders/workflows/python_uv/validator.py Outdated Show resolved Hide resolved

sriram-mv force-pushed the python-uv branch 3 times, most recently from ec932ce to 84f5c13 Compare June 21, 2025 04:45

sriram-mv marked this pull request as ready for review June 23, 2025 19:28

sriram-mv requested a review from a team as a code owner June 23, 2025 19:28

Vandita2020 reviewed Jun 24, 2025

View reviewed changes

reedham-aws mentioned this pull request Sep 16, 2025

Feature request: Add uv Support #743

Open

reedham-aws reviewed Sep 25, 2025

View reviewed changes

sriram-mv force-pushed the python-uv branch from 2b19f9e to adeb8d8 Compare October 3, 2025 19:49

sriram-mv added 3 commits October 3, 2025 12:50

tests: additional functional tests for python_uv

c748595

fix(): address comments

ccc51fb

sriram-mv force-pushed the python-uv branch from adeb8d8 to ccc51fb Compare October 3, 2025 19:52

		MESSAGE = "Lock file operation failed: {reason}"


		class ManifestNotFoundError(LambdaBuilderError):

		MESSAGE = "Failed to install dependencies using uv: {reason}"


		class UvResolutionError(LambdaBuilderError):

	try:
	target_artifact_dir = self.artifacts_dir
	# if dependencies folder is provided, download the dependencies into dependencies folder
	if self.dependencies_dir:
	target_artifact_dir = self.dependencies_dir

	package_builder.build_dependencies(
	artifacts_dir_path=target_artifact_dir,
	scratch_dir_path=self.scratch_dir,
	requirements_path=self.manifest_path,
	)
	except PackagerError as ex:
	raise ActionFailedError(str(ex))

	def _extract_python_version(self, runtime: str) -> str:
	def _extract_python_version(self, runtime: Optional[str]) -> str:

		raise ActionFailedError(f"UV build failed: {str(ex)}")


		class CopyDependenciesAction(BaseAction):

feat: Python UV workflow #756

Are you sure you want to change the base?

feat: Python UV workflow #756

Uh oh!

Conversation

sriram-mv commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pkarakal commented Sep 5, 2025

Uh oh!

AdrianCarreno commented Sep 15, 2025

Uh oh!

reedham-aws commented Sep 16, 2025

Uh oh!

sriram-mv commented Sep 21, 2025

Uh oh!

reedham-aws left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sriram-mv commented Jun 20, 2025 •

edited

Loading