ci: add workflow for lambda layer publish and yank #870
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # Lambda Layers Standard Operating Procedures (SOP) | ||
|
|
||
| ## Overview | ||
|
|
||
| This document defines the standard operating procedures for managing Strands Agents Lambda layers across all AWS regions, Python versions, and architectures. | ||
|
|
||
| **Total: 136 individual Lambda layers** (17 regions × 2 architectures × 4 Python versions). All variants must maintain the same layer version number for each PyPI package version, with only one row per PyPI version appearing in documentation. | ||
|
|
||
| ## Deployment Process | ||
|
|
||
| ### 1. Initial Deployment | ||
| 1. Run workflow with ALL options selected (default) | ||
| 2. Specify PyPI package version | ||
| 3. Type "Create Lambda Layer {package_version}" to confirm | ||
| 4. All 136 individual layers deploy in parallel (4 Python × 2 arch × 17 regions) | ||
| 5. Each layer gets its own unique name: `strands-agents-py{PYTHON_VERSION}-{ARCH}` | ||
|
|
||
| ### 2. Version Buffering for New Variants | ||
| When adding new variants (new Python version, architecture, or region): | ||
|
|
||
| 1. **Determine target layer version**: Check existing variants to find the highest layer version | ||
| 2. **Buffer deployment**: Deploy new variants multiple times until layer version matches existing variants | ||
| 3. **Example**: If existing variants are at layer version 5, deploy new variant 5 times to reach version 5 | ||
|
There was a problem hiding this comment. Why do we need to buffer deployments? If we add a new variant, no one would could be depending on versions before that one. I would like to avoid backfilling versions if possible |
||
|
|
||
| ### 3. Handling Transient Failures | ||
| When some regions fail during deployment: | ||
|
|
||
| 1. **Identify failed regions**: Check which combinations didn't complete successfully | ||
| 2. **Targeted redeployment**: Use specific region/arch/Python inputs to redeploy failed combinations | ||
| 3. **Version alignment**: Continue deploying until all variants reach the same layer version | ||
| 4. **Verification**: Confirm all combinations have identical layer versions before updating docs | ||
|
|
||
| ## Yank Process | ||
|
|
||
| ### Yank Procedure | ||
| 1. Use the `yank_lambda_layer` GitHub action workflow | ||
| 2. Specify the layer version to yank | ||
| 3. Type "Yank Lambda Layer {layer_version}" to confirm | ||
| 4. **Full yank**: Run with ALL options selected (default) to yank all 136 variants OR **Partial yank**: Specify Python versions, architectures, and regions for targeted yanking | ||
| 6. Update documentation | ||
| 7. **Communication**: Notify users through appropriate channels | ||
|
|
||
| **Note**: Yanking deletes layer versions completely. Existing Lambda functions using the layer continue to work, but new functions cannot use the yanked version. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,198 @@ | ||
| name: Publish PyPI Package to Lambda Layer | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| package_version: | ||
| description: 'Package version to download' | ||
| required: true | ||
| type: string | ||
| python_version: | ||
| description: 'Python version' | ||
| required: true | ||
| default: 'ALL' | ||
| type: choice | ||
| options: ['ALL', '3.10', '3.11', '3.12', '3.13'] | ||
| architecture: | ||
| description: 'Architecture' | ||
| required: true | ||
| default: 'ALL' | ||
| type: choice | ||
| options: ['ALL', 'x86_64', 'aarch64'] | ||
| region: | ||
| description: 'AWS region' | ||
| required: true | ||
| default: 'ALL' | ||
| type: choice | ||
| # Only non opt-in regions included for now | ||
| options: ['ALL', 'us-east-1', 'us-east-2', 'us-west-1', 'us-west-2', 'ap-south-1', 'ap-northeast-1', 'ap-northeast-2', 'ap-northeast-3', 'ap-southeast-1', 'ap-southeast-2', 'ca-central-1', 'eu-central-1', 'eu-west-1', 'eu-west-2', 'eu-west-3', 'eu-north-1', 'sa-east-1'] | ||
| confirm: | ||
| description: 'Type "Create Lambda Layer {PyPI version}" to confirm publishing the layer' | ||
| required: true | ||
| type: string | ||
|
|
||
| env: | ||
| BUCKET_NAME: strands-agents-lambda-layer | ||
|
|
||
| jobs: | ||
| validate: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Validate confirmation | ||
|
|
||
| run: | | ||
| CONFIRM="${{ inputs.confirm }}" | ||
| EXPECTED="Create Lambda Layer ${{ inputs.package_version }}" | ||
| if [ "$CONFIRM" != "$EXPECTED" ]; then | ||
| echo "Confirmation failed. You must type exactly '$EXPECTED' to proceed." | ||
| exit 1 | ||
| fi | ||
| echo "Confirmation validated" | ||
|
|
||
| create-buckets: | ||
| needs: validate | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| matrix: | ||
| region: ${{ inputs.region == 'ALL' && fromJson('["us-east-1", "us-east-2", "us-west-1", "us-west-2", "ap-south-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-southeast-1", "ap-southeast-2", "ca-central-1", "eu-central-1", "eu-west-1", "eu-west-2", "eu-west-3", "eu-north-1", "sa-east-1"]') || fromJson(format('["{0}"]', inputs.region)) }} | ||
| permissions: | ||
| id-token: write | ||
| steps: | ||
| - name: Configure AWS credentials | ||
| uses: aws-actions/configure-aws-credentials@v4 | ||
| with: | ||
| role-to-assume: ${{ secrets.STRANDS_LAMBDA_LAYER_PUBLISHER_ROLE }} | ||
| aws-region: ${{ matrix.region }} | ||
|
|
||
| - name: Create S3 bucket | ||
| run: | | ||
| REGION="${{ matrix.region }}" | ||
| ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) | ||
| REGIONAL_BUCKET="${{ env.BUCKET_NAME }}-${ACCOUNT_ID}-${REGION}" | ||
|
|
||
| if ! aws s3api head-bucket --bucket "$REGIONAL_BUCKET" 2>/dev/null; then | ||
| if [ "$REGION" = "us-east-1" ]; then | ||
| aws s3api create-bucket --bucket "$REGIONAL_BUCKET" --region "$REGION" 2>/dev/null || echo "Bucket $REGIONAL_BUCKET already exists" | ||
| else | ||
| aws s3api create-bucket --bucket "$REGIONAL_BUCKET" --region "$REGION" --create-bucket-configuration LocationConstraint="$REGION" 2>/dev/null || echo "Bucket $REGIONAL_BUCKET already exists" | ||
| fi | ||
| echo "S3 bucket ready: $REGIONAL_BUCKET" | ||
| else | ||
| echo "S3 bucket already exists: $REGIONAL_BUCKET" | ||
| fi | ||
|
|
||
| package-and-upload: | ||
| needs: create-buckets | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| matrix: | ||
| python-version: ${{ inputs.python_version == 'ALL' && fromJson('["3.10", "3.11", "3.12", "3.13"]') || fromJson(format('["{0}"]', inputs.python_version)) }} | ||
| architecture: ${{ inputs.architecture == 'ALL' && fromJson('["x86_64", "aarch64"]') || fromJson(format('["{0}"]', inputs.architecture)) }} | ||
| region: ${{ inputs.region == 'ALL' && fromJson('["us-east-1", "us-east-2", "us-west-1", "us-west-2", "ap-south-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-southeast-1", "ap-southeast-2", "ca-central-1", "eu-central-1", "eu-west-1", "eu-west-2", "eu-west-3", "eu-north-1", "sa-east-1"]') || fromJson(format('["{0}"]', inputs.region)) }} | ||
|
There was a problem hiding this comment. Why would we ever not deploy to all regions? |
||
|
|
||
| permissions: | ||
| id-token: write | ||
|
|
||
| steps: | ||
| - name: Set up Python | ||
| uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
|
|
||
| - name: Configure AWS credentials | ||
| uses: aws-actions/configure-aws-credentials@v4 | ||
| with: | ||
| role-to-assume: ${{ secrets.STRANDS_LAMBDA_LAYER_PUBLISHER_ROLE }} | ||
| aws-region: ${{ matrix.region }} | ||
|
|
||
| - name: Create layer directory structure | ||
| run: | | ||
| mkdir -p layer/python | ||
|
|
||
| - name: Download and install package | ||
| run: | | ||
| pip install strands-agents==${{ inputs.package_version }} \ | ||
|
There was a problem hiding this comment. Can we print out and record the dependency versions for each lambda layer? |
||
| --python-version ${{ matrix.python-version }} \ | ||
| --platform manylinux2014_${{ matrix.architecture }} \ | ||
| -t layer/python/ \ | ||
| --only-binary=:all: | ||
|
|
||
| - name: Create layer zip | ||
| run: | | ||
| cd layer | ||
| zip -r ../lambda-layer.zip . | ||
|
|
||
| - name: Upload to S3 | ||
| run: | | ||
| PYTHON_VERSION="${{ matrix.python-version }}" | ||
| ARCH="${{ matrix.architecture }}" | ||
| REGION="${{ matrix.region }}" | ||
| LAYER_NAME="strands-agents-py${PYTHON_VERSION//./_}-${ARCH}" | ||
| ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) | ||
| BUCKET_NAME="${{ env.BUCKET_NAME }}-${ACCOUNT_ID}-${REGION}" | ||
| LAYER_KEY="$LAYER_NAME/v${{ inputs.package_version }}/lambda-layer.zip" | ||
|
There was a problem hiding this comment. What if we need to re-release a layer version with the same strands package version? Will this key collide? Can we infer what the new layer version will be, and add that to the key name here? |
||
|
|
||
| aws s3 cp lambda-layer.zip "s3://$BUCKET_NAME/$LAYER_KEY" --region "$REGION" | ||
| echo "Uploaded layer to s3://$BUCKET_NAME/$LAYER_KEY" | ||
|
|
||
| publish-layer: | ||
| needs: package-and-upload | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| matrix: | ||
| python-version: ${{ inputs.python_version == 'ALL' && fromJson('["3.10", "3.11", "3.12", "3.13"]') || fromJson(format('["{0}"]', inputs.python_version)) }} | ||
| architecture: ${{ inputs.architecture == 'ALL' && fromJson('["x86_64", "aarch64"]') || fromJson(format('["{0}"]', inputs.architecture)) }} | ||
| region: ${{ inputs.region == 'ALL' && fromJson('["us-east-1", "us-east-2", "us-west-1", "us-west-2", "ap-south-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-southeast-1", "ap-southeast-2", "ca-central-1", "eu-central-1", "eu-west-1", "eu-west-2", "eu-west-3", "eu-north-1", "sa-east-1"]') || fromJson(format('["{0}"]', inputs.region)) }} | ||
|
|
||
| permissions: | ||
| id-token: write | ||
|
|
||
| steps: | ||
| - name: Configure AWS credentials | ||
| uses: aws-actions/configure-aws-credentials@v4 | ||
| with: | ||
| role-to-assume: ${{ secrets.STRANDS_LAMBDA_LAYER_PUBLISHER_ROLE }} | ||
| aws-region: ${{ matrix.region }} | ||
|
|
||
| - name: Publish layer | ||
| run: | | ||
| PYTHON_VERSION="${{ matrix.python-version }}" | ||
| ARCH="${{ matrix.architecture }}" | ||
| REGION="${{ matrix.region }}" | ||
| LAYER_NAME="strands-agents-py${PYTHON_VERSION//./_}-${ARCH}" | ||
| ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) | ||
| REGION_BUCKET="${{ env.BUCKET_NAME }}-${ACCOUNT_ID}-${REGION}" | ||
| LAYER_KEY="$LAYER_NAME/v${{ inputs.package_version }}/lambda-layer.zip" | ||
|
|
||
| DESCRIPTION="PyPI package: strands-agents v${{ inputs.package_version }} (Python $PYTHON_VERSION, $ARCH)" | ||
|
|
||
| # Set compatible architecture based on matrix architecture | ||
| if [ "$ARCH" = "x86_64" ]; then | ||
| COMPATIBLE_ARCH="x86_64" | ||
| else | ||
| COMPATIBLE_ARCH="arm64" | ||
| fi | ||
|
|
||
| LAYER_OUTPUT=$(aws lambda publish-layer-version \ | ||
| --layer-name $LAYER_NAME \ | ||
| --description "$DESCRIPTION" \ | ||
| --content S3Bucket=$REGION_BUCKET,S3Key=$LAYER_KEY \ | ||
| --compatible-runtimes python${{ matrix.python-version }} \ | ||
| --compatible-architectures $COMPATIBLE_ARCH \ | ||
| --region "$REGION" \ | ||
| --license-info Apache-2.0 \ | ||
| --output json) | ||
|
|
||
| LAYER_ARN=$(echo "$LAYER_OUTPUT" | jq -r '.LayerArn') | ||
| LAYER_VERSION=$(echo "$LAYER_OUTPUT" | jq -r '.Version') | ||
|
|
||
| echo "Published layer version $LAYER_VERSION with ARN: $LAYER_ARN in region $REGION" | ||
|
|
||
| aws lambda add-layer-version-permission \ | ||
| --layer-name $LAYER_NAME \ | ||
| --version-number $LAYER_VERSION \ | ||
| --statement-id public \ | ||
| --action lambda:GetLayerVersion \ | ||
| --principal '*' \ | ||
| --region "$REGION" | ||
|
|
||
| echo "Successfully published layer version $LAYER_VERSION in region $REGION" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| name: Yank Lambda Layer | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| layer_version: | ||
| description: 'Layer version to yank' | ||
| required: true | ||
| type: string | ||
| python_version: | ||
| description: 'Python version' | ||
| required: true | ||
| default: 'ALL' | ||
| type: choice | ||
| options: ['ALL', '3.10', '3.11', '3.12', '3.13'] | ||
| architecture: | ||
| description: 'Architecture' | ||
| required: true | ||
| default: 'ALL' | ||
| type: choice | ||
| options: ['ALL', 'x86_64', 'aarch64'] | ||
| region: | ||
| description: 'AWS region' | ||
| required: true | ||
| default: 'ALL' | ||
| type: choice | ||
| # Only non opt-in regions included for now | ||
| options: ['ALL', 'us-east-1', 'us-east-2', 'us-west-1', 'us-west-2', 'ap-south-1', 'ap-northeast-1', 'ap-northeast-2', 'ap-northeast-3', 'ap-southeast-1', 'ap-southeast-2', 'ca-central-1', 'eu-central-1', 'eu-west-1', 'eu-west-2', 'eu-west-3', 'eu-north-1', 'sa-east-1'] | ||
| confirm: | ||
| description: 'Type "Yank Lambda Layer {layer version}" to confirm yanking the layer' | ||
| required: true | ||
| type: string | ||
|
|
||
| jobs: | ||
| yank-layer: | ||
| runs-on: ubuntu-latest | ||
| continue-on-error: true | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| python-version: ${{ inputs.python_version == 'ALL' && fromJson('["3.10", "3.11", "3.12", "3.13"]') || fromJson(format('["{0}"]', inputs.python_version)) }} | ||
| architecture: ${{ inputs.architecture == 'ALL' && fromJson('["x86_64", "aarch64"]') || fromJson(format('["{0}"]', inputs.architecture)) }} | ||
| region: ${{ inputs.region == 'ALL' && fromJson('["us-east-1", "us-east-2", "us-west-1", "us-west-2", "ap-south-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-southeast-1", "ap-southeast-2", "ca-central-1", "eu-central-1", "eu-west-1", "eu-west-2", "eu-west-3", "eu-north-1", "sa-east-1"]') || fromJson(format('["{0}"]', inputs.region)) }} | ||
|
|
||
| permissions: | ||
| id-token: write | ||
|
|
||
| steps: | ||
| - name: Validate confirmation | ||
| run: | | ||
| CONFIRM="${{ inputs.confirm }}" | ||
| EXPECTED="Yank Lambda Layer ${{ inputs.layer_version }}" | ||
| if [ "$CONFIRM" != "$EXPECTED" ]; then | ||
| echo "Confirmation failed. You must type exactly '$EXPECTED' to proceed." | ||
| exit 1 | ||
| fi | ||
| echo "Confirmation validated" | ||
|
|
||
| - name: Configure AWS credentials | ||
| uses: aws-actions/configure-aws-credentials@v4 | ||
| with: | ||
| role-to-assume: ${{ secrets.STRANDS_LAMBDA_LAYER_PUBLISHER_ROLE }} | ||
| aws-region: ${{ matrix.region }} | ||
|
|
||
| - name: Yank layer | ||
| run: | | ||
| PYTHON_VERSION="${{ matrix.python-version }}" | ||
| ARCH="${{ matrix.architecture }}" | ||
| REGION="${{ matrix.region }}" | ||
| LAYER_NAME="strands-agents-py${PYTHON_VERSION//./_}-${ARCH}" | ||
| LAYER_VERSION="${{ inputs.layer_version }}" | ||
|
|
||
| echo "Attempting to yank layer $LAYER_NAME version $LAYER_VERSION in region $REGION" | ||
|
|
||
| # Delete the layer version completely | ||
| aws lambda delete-layer-version \ | ||
| --layer-name $LAYER_NAME \ | ||
| --version-number $LAYER_VERSION \ | ||
| --region "$REGION" | ||
|
|
||
| echo "Completed yank attempt for layer $LAYER_NAME version $LAYER_VERSION in region $REGION" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be nice to note the exact versions of dependencies included in the layer, in case certain dependency versions come with quirks or bugs. We should also note the version of strands being released in each layer.