diff --git a/aws-step-functions-jsonata/POWER.md b/aws-step-functions-jsonata/POWER.md new file mode 100644 index 0000000..b46765d --- /dev/null +++ b/aws-step-functions-jsonata/POWER.md @@ -0,0 +1,183 @@ +--- +name: "step-functions-jsonata" +displayName: "AWS Step Functions with JSONata" +description: "Build AWS Step Functions state machines using JSONata query language. Covers ASL structure, all state types, variables, data transformation, error handling, and service integrations in JSONata mode." +keywords: ["step functions", "state machine", "serverless", "jsonata", "asl", "amazon states language", "workflow orchestration"] +author: "Jeff Palmer https://linkedin.com/in/jeffrey-palmer/" +--- + +# Step Functions JSONata + +Build AWS Step Functions state machines using the JSONata query language instead of legacy JSONPath. JSONata simplifies data transformation, reduces boilerplate, and reduces external dependencies. + +## Overview + +AWS Step Functions uses Amazon States Language (ASL) to define state machines as JSON. With JSONata mode, you replace the five JSONPath I/O fields (InputPath, Parameters, ResultSelector, ResultPath, OutputPath) with just two fields: `Arguments` and `Output`. You also gain workflow variables via `Assign`, and powerful `Condition` expressions in Choice states. + +This power provides comprehensive guidance for writing state machines in JSONata mode, covering: +- ASL structure and all eight state types in JSONata mode +- The `$states` reserved variable and JSONata expression syntax +- Workflow variables with `Assign` for cross-state data sharing +- Data transformation patterns with `Arguments` and `Output` +- Error handling with `Retry` and `Catch` +- Service integration patterns (Lambda, DynamoDB, SNS, SQS, etc.) + +## When to Load Steering Files + +Load the appropriate steering file based on what the user is working on: + +- **ASL structure**, **state types**, **Task**, **Pass**, **Choice**, **Wait**, **Succeed**, **Fail**, **Parallel**, **Map** → see `asl-state-types.md` +- **Variables**, **Assign**, **data passing**, **scope**, **$states**, **input**, **output**, **Arguments**, **Output**, **data transformation** → see `variables-and-data.md` +- **Error handling**, **Retry**, **Catch**, **fallback**, **error codes**, **States.Timeout**, **States.ALL** → see `error-handling.md` +- **Service integrations**, **Lambda invoke**, **DynamoDB**, **SNS**, **SQS**, **SDK integrations**, **Resource ARN**, **sync**, **async** → see `service-integrations.md` + +## Quick Reference + +### Enabling JSONata + +Set `QueryLanguage` at the top level to apply to all states: + +```json +{ + "QueryLanguage": "JSONata", + "StartAt": "MyState", + "States": { ... } +} +``` + +### JSONata Expression Syntax + +Wrap expressions in `{% %}`: + +```json +"Output": "{% $states.input.customer.name %}" +"TimeoutSeconds": "{% $timeout %}" +"Condition": "{% $states.input.age >= 18 %}" +``` + +### The `$states` Reserved Variable + +``` +$states.input → Original state input +$states.result → Task/Parallel/Map result (on success) +$states.errorOutput → Error output (only in Catch) +$states.context → Execution context object +``` + +### Key Fields in JSONata Mode + +| Field | Purpose | Available In | +|-------|---------|-------------| +| `Arguments` | Input to task/branches | Task, Parallel | +| `Output` | Transform state output | All except Fail | +| `Assign` | Store workflow variables | All except Succeed, Fail | +| `Condition` | Boolean branching | Choice rules | +| `Items` | Array for iteration | Map | + +### JSONata Functions Provided by Step Functions + +| Function | Purpose | +|----------|---------| +| `$partition(array, size)` | Partition array into chunks | +| `$range(start, end, step)` | Generate array of values | +| `$hash(data, algorithm)` | Calculate hash (MD5, SHA-1, SHA-256, SHA-384, SHA-512) | +| `$random([seed])` | Random number 0 ≤ n < 1, optional seed | +| `$uuid()` | Generate v4 UUID | +| `$parse(jsonString)` | Deserialize JSON string | + +Plus all [built-in JSONata functions](https://github.com/jsonata-js/jsonata/tree/master/docs) + +### Minimal Complete Example + +```json +{ + "Comment": "Order processing workflow", + "QueryLanguage": "JSONata", + "StartAt": "ValidateOrder", + "States": { + "ValidateOrder": { + "Type": "Task", + "Resource": "arn:aws:states:::dynamodb:getItem", + "Arguments": { + "TableName": "OrdersTable", + "Key": { + "orderId": { + "S": "{% $states.input.orderId %}" + } + } + }, + "Assign": { + "orderId": "{% $states.input.orderId %}" + }, + "Output": "{% $states.result.Item %}", + "Next": "CheckStock" + }, + "CheckStock": { + "Type": "Choice", + "Choices": [ + { + "Condition": "{% $states.input.inStock = true %}", + "Next": "ProcessPayment" + } + ], + "Default": "OutOfStock" + }, + "ProcessPayment": { + "Type": "Task", + "Resource": "arn:aws:states:::sqs:sendMessage", + "Arguments": { + "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/PaymentQueue", + "MessageBody": "{% $string({'orderId': $orderId, 'amount': $states.input.total.N}) %}" + }, + "Output": { + "orderId": "{% $orderId %}", + "messageId": "{% $states.result.MessageId %}" + }, + "Retry": [ + { + "ErrorEquals": ["States.TaskFailed"], + "IntervalSeconds": 2, + "MaxAttempts": 3, + "BackoffRate": 2.0 + } + ], + "End": true + }, + "OutOfStock": { + "Type": "Fail", + "Error": "OutOfStockError", + "Cause": "Requested item is out of stock" + } + } +} +``` + +## Best Practices + +- Always set `"QueryLanguage": "JSONata"` at the top level for new state machines +- Use `Assign` to store data needed in later states instead of threading it through Output +- Keep `Output` minimal — only provide what the next state actually needs +- Use `$states.input` to reference original state input, not `$` (which is restricted at top level in JSONata) +- Remember: `Assign` and `Output` are evaluated in parallel — variable assignments in `Assign` are NOT available in `Output` of the same state +- All JSONata expressions must produce a defined value — `$data.nonExistentField` throws `States.QueryEvaluationError` +- Use `$states.context.Execution.Input` to access the original workflow input from any state +- Save state machine definitions with `.asl.json` extension when working outside the console +- Prefer the optimized Lambda integration (`arn:aws:states:::lambda:invoke`) over the SDK integration + +## Troubleshooting + +### Common Errors + +- `States.QueryEvaluationError` — JSONata expression failed. Check for type errors, undefined fields, or out-of-range values. +- Mixing JSONPath fields (`Parameters`, `InputPath`, `ResultPath`, etc.) with JSONata `QueryLanguage` — these are mutually exclusive. +- Using `$` or `$$` at the top level of a JSONata expression — use `$states.input` instead. +- Forgetting `{% %}` delimiters around JSONata expressions — the string will be treated as a literal. +- Assigning variables in `Assign` and expecting them in `Output` of the same state — new values only take effect in the next state. + +## Resources + +- [ASL Specification](https://states-language.net/spec.html) +- [Transforming data with JSONata in Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/transforming-data.html) +- [Passing data between states with variables](https://docs.aws.amazon.com/step-functions/latest/dg/workflow-variables.html) +- [JSONata documentation](https://docs.jsonata.org/overview.html) +- [Step Functions Developer Guide](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) diff --git a/aws-step-functions-jsonata/steering/architecture-patterns.md b/aws-step-functions-jsonata/steering/architecture-patterns.md new file mode 100644 index 0000000..9f82f6a --- /dev/null +++ b/aws-step-functions-jsonata/steering/architecture-patterns.md @@ -0,0 +1,488 @@ +# Architecture Patterns (JSONata Mode) + +## Polling Loop (Wait → Check → Choice) + +Many AWS operations are asynchronous — you start them and then poll until they complete. The pattern is: initial wait → call describe/status API → check result → short wait → loop back. + +```json +"SubmitOrder": { + "Type": "Task", + "Resource": "arn:aws:states:::sqs:sendMessage", + "Arguments": { + "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/FulfillmentQueue", + "MessageBody": "{% $string({'orderId': $orderId, 'items': $states.input.items}) %}" + }, + "Assign": { "fulfillmentOrderId": "{% $orderId %}" }, + "Next": "InitialWaitForFulfillment" +}, +"InitialWaitForFulfillment": { + "Type": "Wait", + "Seconds": 300, + "Next": "CheckFulfillmentStatus" +}, +"CheckFulfillmentStatus": { + "Type": "Task", + "Resource": "arn:aws:states:::dynamodb:getItem", + "Arguments": { + "TableName": "OrdersTable", + "Key": { "orderId": { "S": "{% $fulfillmentOrderId %}" } } + }, + "Assign": { "orderStatus": "{% $states.result.Item.status.S %}" }, + "Next": "EvaluateFulfillment", + "Retry": [ + { "ErrorEquals": ["States.TaskFailed", "ThrottlingException"], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2 } + ] +}, +"EvaluateFulfillment": { + "Type": "Choice", + "Choices": [ + { "Condition": "{% $orderStatus = 'fulfilled' %}", "Next": "FulfillmentComplete" }, + { "Condition": "{% $orderStatus in ['failed', 'cancelled'] %}", "Next": "FulfillmentFailed" } + ], + "Default": "WaitBeforeNextPoll" +}, +"WaitBeforeNextPoll": { + "Type": "Wait", + "Seconds": 60, + "Next": "CheckFulfillmentStatus" +} +``` + +Key elements: +- Initial longer wait gives the operation time to start. Shorter poll interval for subsequent checks. +- Choice state routes to success, failure, or back to the wait loop. +- Always add Retry on the status-check Task to handle transient API errors. +- Consider adding `TimeoutSeconds` on the state machine or a counter variable to prevent infinite polling. + +--- + +## Compensation / Saga Pattern + +Step Functions has no built-in rollback. The saga pattern chains compensating actions in reverse order. Each forward step has a Catch that records which step failed, then routes to the appropriate compensation entry point. + +```json +"ReserveInventory": { + "Type": "Task", + "Resource": "arn:aws:states:::dynamodb:updateItem", + "Arguments": { + "TableName": "InventoryTable", + "Key": { "productId": { "S": "{% $states.input.productId %}" } }, + "UpdateExpression": "SET reserved = reserved + :qty", + "ExpressionAttributeValues": { ":qty": { "N": "{% $string($states.input.quantity) %}" } } + }, + "Assign": { "reservedQty": "{% $states.input.quantity %}" }, + "Catch": [ + { "ErrorEquals": ["States.ALL"], "Assign": { "failedStep": "ReserveInventory", "errorInfo": "{% $states.errorOutput %}" }, "Next": "OrderFailed" } + ], + "Next": "ChargePayment" +}, +"ChargePayment": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:ChargeCard:$LATEST", + "Payload": { "orderId": "{% $orderId %}", "amount": "{% $states.input.total %}" } + }, + "Assign": { "chargeId": "{% $states.result.Payload.chargeId %}" }, + "Output": "{% $states.result.Payload %}", + "Catch": [ + { "ErrorEquals": ["States.ALL"], "Assign": { "failedStep": "ChargePayment", "errorInfo": "{% $states.errorOutput %}" }, "Next": "ReleaseInventory" } + ], + "Next": "ShipOrder" +}, +"ShipOrder": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:ShipOrder:$LATEST", + "Payload": { "orderId": "{% $orderId %}" } + }, + "Catch": [ + { "ErrorEquals": ["States.ALL"], "Assign": { "failedStep": "ShipOrder", "errorInfo": "{% $states.errorOutput %}" }, "Next": "RefundPayment" } + ], + "Next": "OrderComplete" +}, +"RefundPayment": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:RefundCharge:$LATEST", + "Payload": { "chargeId": "{% $chargeId %}", "reason": "{% $errorInfo.Cause %}" } + }, + "Next": "ReleaseInventory" +}, +"ReleaseInventory": { + "Type": "Task", + "Resource": "arn:aws:states:::dynamodb:updateItem", + "Arguments": { + "TableName": "InventoryTable", + "Key": { "productId": { "S": "{% $states.input.productId %}" } }, + "UpdateExpression": "SET reserved = reserved - :qty", + "ExpressionAttributeValues": { ":qty": { "N": "{% $string($reservedQty) %}" } } + }, + "Next": "OrderFailed" +}, +"OrderFailed": { + "Type": "Fail", + "Error": "{% $failedStep & 'Error' %}", + "Cause": "{% 'Order ' & $orderId & ' failed at ' & $failedStep & ': ' & ($exists($errorInfo.Cause) ? $errorInfo.Cause : 'Unknown') %}" +} +``` + +Compensation chain: `ReserveInventory` fails → `OrderFailed`. `ChargePayment` fails → `ReleaseInventory` → `OrderFailed`. `ShipOrder` fails → `RefundPayment` → `ReleaseInventory` → `OrderFailed`. Each Catch records `$failedStep` and `$errorInfo`. Compensation states use variables from forward steps (`$chargeId`, `$reservedQty`) to know what to undo. + +--- + +## Nested Map / Parallel Structures + +Map, Parallel, and Task states nest in any combination. The key constraint is understanding variable scope and data flow at each nesting boundary. + +```json +"ProcessAllOrders": { + "Type": "Map", + "Items": "{% $states.input.orders %}", + "MaxConcurrency": 5, + "ItemProcessor": { + "ProcessorConfig": { "Mode": "INLINE" }, + "StartAt": "ProcessSingleOrder", + "States": { + "ProcessSingleOrder": { + "Type": "Parallel", + "Branches": [ + { + "StartAt": "ValidatePayment", + "States": { + "ValidatePayment": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:ValidatePayment:$LATEST", + "Payload": "{% $states.input %}" + }, + "Output": "{% $states.result.Payload %}", + "End": true + } + } + }, + { + "StartAt": "CheckInventory", + "States": { + "CheckInventory": { + "Type": "Task", + "Resource": "arn:aws:states:::dynamodb:getItem", + "Arguments": { + "TableName": "InventoryTable", + "Key": { "productId": { "S": "{% $states.input.productId %}" } } + }, + "Output": "{% $states.result.Item %}", + "End": true + } + } + } + ], + "Output": { "payment": "{% $states.result[0] %}", "inventory": "{% $states.result[1] %}" }, + "End": true + } + } + }, + "Assign": { "orderResults": "{% $states.result %}" }, + "Next": "Summarize" +} +``` + +### Variable Scoping Across Nesting Levels + +Each nesting level creates a new scope. Inner scopes can READ outer variables but CANNOT ASSIGN to them — use `Output` on terminal states to pass data back up. Parallel branches and Map iterations are isolated from each other. Variable names must be unique across all nesting levels (no shadowing). Exception: Distributed Map (`"Mode": "DISTRIBUTED"`) cannot read outer scope variables at all. + +Data flows down via state input (use `ItemSelector` for Map, `Arguments` for Parallel) and up via `Output` on terminal states. Parallel result is an array per branch; Map result is an array per iteration. + +--- + +## Scatter-Gather with Partial Results + +When calling unreliable external APIs per-item, use `ToleratedFailurePercentage` on a Map to continue with whatever succeeded, then post-process the results to separate successes from failures. Failed iterations return objects with `Error` and `Cause` fields. + +```json +"CallExternalAPIs": { + "Type": "Map", + "Items": "{% $states.input.records %}", + "MaxConcurrency": 10, + "ToleratedFailurePercentage": 100, + "ItemProcessor": { + "ProcessorConfig": { "Mode": "INLINE" }, + "StartAt": "CallAPI", + "States": { + "CallAPI": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:CallExternalAPI:$LATEST", + "Payload": "{% $states.input %}" + }, + "Output": "{% $states.result.Payload %}", + "Retry": [ + { "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 2, "MaxAttempts": 2, "BackoffRate": 2.0, "JitterStrategy": "FULL" } + ], + "End": true + } + } + }, + "Next": "SplitResults" +}, +"SplitResults": { + "Type": "Pass", + "Assign": { + "successes": "{% ( $s := $states.input[$not($exists(Error))]; $type($s) = 'array' ? $s : $exists($s) ? [$s] : [] ) %}", + "failures": "{% ( $f := $states.input[$exists(Error)]; $type($f) = 'array' ? $f : $exists($f) ? [$f] : [] ) %}" + }, + "Output": { + "successes": "{% ( $s := $states.input[$not($exists(Error))]; $type($s) = 'array' ? $s : $exists($s) ? [$s] : [] ) %}", + "failures": "{% ( $f := $states.input[$exists(Error)]; $type($f) = 'array' ? $f : $exists($f) ? [$f] : [] ) %}", + "totalProcessed": "{% $count($states.input) %}" + }, + "Next": "EvaluateResults" +}, +"EvaluateResults": { + "Type": "Choice", + "Choices": [ + { "Condition": "{% $count($successes) = 0 %}", "Next": "AllFailed" } + ], + "Default": "ProcessSuccesses" +} +``` + +Key elements: +- `ToleratedFailurePercentage: 100` lets the Map complete even if every item fails. Lower the threshold to bail out early. +- Filter on `$exists(Error)` to separate failed from successful iterations. +- Guard filtered results with the `$type`/`$exists`/`[]` pattern — JSONata returns a single object (not a 1-element array) when exactly one item matches, and undefined when nothing matches. + +--- + +## Semaphore / Concurrency Lock + +Step Functions has no native mutual exclusion. Use DynamoDB conditional writes as a distributed lock when only one execution should process a given resource at a time. Pattern: acquire lock → do work → release lock, with Catch ensuring release on failure. + +```json +"AcquireLock": { + "Type": "Task", + "Resource": "arn:aws:states:::dynamodb:putItem", + "Arguments": { + "TableName": "LocksTable", + "Item": { + "lockId": { "S": "{% $states.input.customerId %}" }, + "executionId": { "S": "{% $states.context.Execution.Id %}" }, + "expiresAt": { "N": "{% $string($toMillis($now()) + 900000) %}" } + }, + "ConditionExpression": "attribute_not_exists(lockId) OR expiresAt < :now", + "ExpressionAttributeValues": { + ":now": { "N": "{% $string($toMillis($now())) %}" } + } + }, + "Retry": [ + { "ErrorEquals": ["DynamoDB.ConditionalCheckFailedException"], "IntervalSeconds": 5, "MaxAttempts": 12, "BackoffRate": 1.5, "JitterStrategy": "FULL" } + ], + "Catch": [ + { "ErrorEquals": ["DynamoDB.ConditionalCheckFailedException"], "Assign": { "lockError": "{% $states.errorOutput %}" }, "Next": "LockUnavailable" } + ], + "Next": "DoProtectedWork" +}, +"DoProtectedWork": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:ProcessCustomer:$LATEST", + "Payload": "{% $states.input %}" + }, + "Output": "{% $states.result.Payload %}", + "Catch": [ + { "ErrorEquals": ["States.ALL"], "Assign": { "workError": "{% $states.errorOutput %}" }, "Next": "ReleaseLock" } + ], + "Next": "ReleaseLock" +}, +"ReleaseLock": { + "Type": "Task", + "Resource": "arn:aws:states:::dynamodb:deleteItem", + "Arguments": { + "TableName": "LocksTable", + "Key": { "lockId": { "S": "{% $states.input.customerId %}" } }, + "ConditionExpression": "executionId = :execId", + "ExpressionAttributeValues": { ":execId": { "S": "{% $states.context.Execution.Id %}" } } + }, + "Retry": [ + { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 1, "MaxAttempts": 3, "BackoffRate": 2.0 } + ], + "Next": "CheckWorkResult" +}, +"CheckWorkResult": { + "Type": "Choice", + "Choices": [ + { "Condition": "{% $exists($workError) %}", "Next": "WorkFailed" } + ], + "Default": "Done" +}, +"LockUnavailable": { + "Type": "Fail", + "Error": "LockContention", + "Cause": "{% 'Could not acquire lock for ' & $states.input.customerId & ' after retries' %}" +} +``` + +Key elements: +- `ConditionExpression` with `attribute_not_exists` ensures only one writer wins. The `expiresAt` check provides stale-lock recovery if an execution crashes without releasing. +- `executionId` on the lock item lets `ReleaseLock` conditionally delete only its own lock. +- Retry on `ConditionalCheckFailedException` acts as a spin-wait. Tune `MaxAttempts` and `IntervalSeconds` based on expected hold time. +- Catch on `DoProtectedWork` routes to `ReleaseLock` so the lock is always released. After releasing, `CheckWorkResult` re-raises the error path. +- Set `expiresAt` to a reasonable TTL (here 15 min). Use a DynamoDB TTL attribute to auto-clean expired locks. + +--- + +## Human-in-the-Loop with Timeout Escalation + +Chain multiple `.waitForTaskToken` states with `States.Timeout` catches to build escalation: primary approver → manager → auto-reject. + +```json +"RequestApproval": { + "Type": "Task", + "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken", + "Arguments": { + "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/ApprovalQueue", + "MessageBody": "{% $string({'taskToken': $states.context.Task.Token, 'orderId': $orderId, 'approver': $states.input.primaryApprover, 'amount': $states.input.amount}) %}" + }, + "TimeoutSeconds": 86400, + "Assign": { "approvalResult": "{% $states.result %}" }, + "Catch": [ + { "ErrorEquals": ["States.Timeout"], "Assign": { "escalationReason": "Primary approver did not respond within 24 hours" }, "Next": "EscalateToManager" }, + { "ErrorEquals": ["States.ALL"], "Assign": { "approvalError": "{% $states.errorOutput %}" }, "Next": "ApprovalFailed" } + ], + "Next": "EvaluateApproval" +}, +"EscalateToManager": { + "Type": "Task", + "Resource": "arn:aws:states:::sns:publish", + "Arguments": { + "TopicArn": "arn:aws:sns:us-east-1:123456789012:EscalationNotifications", + "Subject": "Approval Escalation", + "Message": "{% 'Order ' & $orderId & ' requires manager approval. ' & $escalationReason %}" + }, + "Next": "WaitForManagerApproval" +}, +"WaitForManagerApproval": { + "Type": "Task", + "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken", + "Arguments": { + "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/ApprovalQueue", + "MessageBody": "{% $string({'taskToken': $states.context.Task.Token, 'orderId': $orderId, 'approver': $states.input.managerApprover, 'amount': $states.input.amount, 'escalated': true}) %}" + }, + "TimeoutSeconds": 43200, + "Assign": { "approvalResult": "{% $states.result %}" }, + "Catch": [ + { + "ErrorEquals": ["States.Timeout"], + "Assign": { "approvalResult": { "decision": "rejected", "reason": "No response from manager within 12 hours — auto-rejected" } }, + "Next": "EvaluateApproval" + }, + { "ErrorEquals": ["States.ALL"], "Assign": { "approvalError": "{% $states.errorOutput %}" }, "Next": "ApprovalFailed" } + ], + "Next": "EvaluateApproval" +}, +"EvaluateApproval": { + "Type": "Choice", + "Choices": [ + { "Condition": "{% $approvalResult.decision = 'approved' %}", "Next": "ProcessApprovedOrder" } + ], + "Default": "OrderRejected" +} +``` + +Key elements: +- Each callback stage has its own `TimeoutSeconds` — shorter for escalation stages since urgency increases. +- `States.Timeout` in Catch distinguishes "no response" from actual errors, routing to the next escalation tier. +- The final tier auto-rejects by assigning a synthetic result in Catch `Assign` and routing to the same `EvaluateApproval` Choice. This avoids duplicating decision logic. +- External system calls `SendTaskSuccess` with `{"decision": "approved"}` or `{"decision": "rejected", "reason": "..."}`. +- Use Standard (not Express) workflows — Express doesn't support `.waitForTaskToken`. + +--- + +## Express → Standard Handoff + +Express workflows are cheaper (pay per request, up to 5 min) but don't support callbacks or long waits. Standard workflows handle those but cost per state transition. Use Express for fast, high-volume ingest and kick off a Standard execution for the long-running tail. + +```json +{ + "Comment": "Express workflow — fast ingest and validation", + "QueryLanguage": "JSONata", + "StartAt": "ValidateInput", + "States": { + "ValidateInput": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:ValidateOrder:$LATEST", + "Payload": "{% $states.input %}" + }, + "Output": "{% $states.result.Payload %}", + "Next": "EnrichData" + }, + "EnrichData": { + "Type": "Parallel", + "Branches": [ + { + "StartAt": "LookupCustomer", + "States": { + "LookupCustomer": { + "Type": "Task", + "Resource": "arn:aws:states:::dynamodb:getItem", + "Arguments": { + "TableName": "CustomersTable", + "Key": { "customerId": { "S": "{% $states.input.customerId %}" } } + }, + "Output": "{% $states.result.Item %}", + "End": true + } + } + }, + { + "StartAt": "LookupPricing", + "States": { + "LookupPricing": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:GetPricing:$LATEST", + "Payload": "{% $states.input %}" + }, + "Output": "{% $states.result.Payload %}", + "End": true + } + } + } + ], + "Output": { + "order": "{% $states.input %}", + "customer": "{% $states.result[0] %}", + "pricing": "{% $states.result[1] %}" + }, + "Next": "HandOffToStandard" + }, + "HandOffToStandard": { + "Type": "Task", + "Resource": "arn:aws:states:::states:startExecution", + "Arguments": { + "StateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:OrderFulfillment-Standard", + "Input": "{% $string($states.input) %}" + }, + "Output": { + "status": "handed_off", + "childExecutionArn": "{% $states.result.ExecutionArn %}" + }, + "End": true + } + } +} +``` + +Key elements: +- Express does validation, enrichment, fan-out — fast, stateless work that benefits from per-request pricing. +- `HandOffToStandard` uses fire-and-forget (no `.sync` suffix) so the Express execution completes immediately. Use `.sync:2` if you need to wait, but watch the 5-minute Express limit. +- Use `$string($states.input)` to serialize — `startExecution` expects a JSON string for `Input`. +- Ideal for event-driven architectures: API Gateway or EventBridge triggers Express at high volume, only orders needing long-running processing incur Standard costs. diff --git a/aws-step-functions-jsonata/steering/asl-state-types.md b/aws-step-functions-jsonata/steering/asl-state-types.md new file mode 100644 index 0000000..c530a49 --- /dev/null +++ b/aws-step-functions-jsonata/steering/asl-state-types.md @@ -0,0 +1,474 @@ +# ASL Structure and State Types (JSONata Mode) + +## State Machine Top-Level Structure + +```json +{ + "Comment": "Description of the state machine", + "QueryLanguage": "JSONata", + "StartAt": "FirstStateName", + "TimeoutSeconds": 3600, + "Version": "1.0", + "States": { + "FirstStateName": { ... }, + "SecondStateName": { ... } + } +} +``` + +- `QueryLanguage`: Set to `"JSONata"` at top level. Defaults to `"JSONPath"` if omitted. +- `StartAt`: Must exactly match a state name (case-sensitive). +- `TimeoutSeconds`: Optional max execution time. Exceeding it throws `States.Timeout`. +- `States`: Required object containing all state definitions. +- State names must be unique and ≤ 80 Unicode characters. + +## Common Fields for All JSONata States + +| Field | Description | +|-------|-------------| +| `Type` | Required. One of: Task, Pass, Choice, Wait, Parallel, Map, Succeed, Fail | +| `Comment` | Optional human-readable description | +| `Next` | Name of next state (required for non-terminal states except Choice) | +| `End` | Set to `true` for terminal states | +| `Output` | Optional. Transform state output. Available in all types except Fail | +| `Assign` | Optional. Store workflow variables. Available in all types except Succeed and Fail | +| `QueryLanguage` | Optional per-state override | + +## Field Availability Matrix (JSONata) + +``` + Task Parallel Map Pass Wait Choice Succeed Fail +Type ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ +Comment ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ +Output ✓ ✓ ✓ ✓ ✓ ✓ ✓ +Assign ✓ ✓ ✓ ✓ ✓ ✓ +Next/End ✓ ✓ ✓ ✓ ✓ +Arguments ✓ ✓ +Retry/Catch ✓ ✓ ✓ +``` + +--- + +## Pass State + +Passes input to output, optionally transforming it. Useful for injecting data or reshaping payloads. + +```json +"InjectData": { + "Type": "Pass", + "Output": { + "greeting": "{% 'Hello, ' & $states.input.name %}", + "timestamp": "{% $now() %}" + }, + "Next": "NextState" +} +``` + +With variable assignment: + +```json +"StoreDefaults": { + "Type": "Pass", + "Assign": { + "retryCount": 0, + "maxRetries": 3, + "config": "{% $states.input.configuration %}" + }, + "Next": "ProcessItem" +} +``` + +Without `Output`, the Pass state copies input to output unchanged. + +--- + +## Task State + +Executes work via AWS service integrations, activities, or HTTP APIs. + +### Required Fields +- `Resource`: ARN identifying the task to execute + +### Optional Fields +- `Arguments`: Input to the task (replaces JSONPath `Parameters`) +- `Output`: Transform the result +- `Assign`: Store variables from input or result +- `TimeoutSeconds`: Max task duration (default 60, accepts JSONata expression) +- `HeartbeatSeconds`: Heartbeat interval (must be < TimeoutSeconds) +- `Retry`: Retry policy array +- `Catch`: Error handler array +- `Credentials`: Cross-account role assumption + +### Lambda Invoke Example + +```json +"InvokeLambda": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:MyFunc:$LATEST", + "Payload": { + "orderId": "{% $states.input.orderId %}", + "customer": "{% $states.input.customer %}" + } + }, + "Assign": { + "processedResult": "{% $states.result.Payload %}" + }, + "Output": "{% $states.result.Payload %}", + "Next": "NextState" +} +``` + +### Dynamic Timeout + +```json +"LongRunningTask": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:SlowFunc:$LATEST", + "Payload": "{% $states.input %}" + }, + "TimeoutSeconds": "{% $states.input.timeoutValue %}", + "HeartbeatSeconds": "{% $states.input.heartbeatValue %}", + "Next": "Done" +} +``` + +--- + +## Choice State + +Adds branching logic. Uses `Condition` field with JSONata boolean expressions (replaces JSONPath `Variable` + comparison operators). + +### Structure + +```json +"RouteOrder": { + "Type": "Choice", + "Choices": [ + { + "Condition": "{% $states.input.orderType = 'express' %}", + "Next": "ExpressShipping" + }, + { + "Condition": "{% $states.input.total > 100 %}", + "Assign": { + "discount": "{% $states.input.total * 0.1 %}" + }, + "Output": { + "total": "{% $states.input.total * 0.9 %}" + }, + "Next": "ApplyDiscount" + }, + { + "Condition": "{% $states.input.priority >= 5 and $states.input.category = 'urgent' %}", + "Next": "PriorityQueue" + } + ], + "Default": "StandardProcessing", + "Assign": { + "routedDefault": true + } +} +``` + +Key points: +- `Condition` must evaluate to a boolean. +- Each Choice Rule can have its own `Assign` and `Output`. +- If a rule matches, its `Assign`/`Output` are used (not the state-level ones). +- If no rule matches, the state-level `Assign` is evaluated and `Default` is followed. +- `Default` is optional but recommended — without it, `States.NoChoiceMatched` is thrown. +- Choice states cannot be terminal (no `End` field). + +### Complex Conditions + +JSONata supports rich boolean logic: + +```json +"Condition": "{% $states.input.age >= 18 and $states.input.age <= 65 %}" +"Condition": "{% $states.input.status = 'active' or $states.input.override = true %}" +"Condition": "{% $not($exists($states.input.error)) %}" +"Condition": "{% $contains($states.input.email, '@') %}" +"Condition": "{% $count($states.input.items) > 0 %}" +"Condition": "{% $states.input.score >= $threshold %}" +``` + +--- + +## Wait State + +Delays execution for a specified duration or until a timestamp. + +### Wait by Seconds + +```json +"WaitTenSeconds": { + "Type": "Wait", + "Seconds": 10, + "Next": "Continue" +} +``` + +### Wait with Dynamic Seconds + +```json +"DynamicWait": { + "Type": "Wait", + "Seconds": "{% $states.input.delaySeconds %}", + "Next": "Continue" +} +``` + +### Wait Until Timestamp + +```json +"WaitUntilDate": { + "Type": "Wait", + "Timestamp": "{% $states.input.scheduledTime %}", + "Next": "Execute" +} +``` + +Timestamps must conform to RFC3339 (e.g., `"2026-03-14T01:59:00Z"`). + +A Wait state must contain exactly one of `Seconds` or `Timestamp`. + +--- + +## Succeed State + +Terminates the state machine (or a Parallel branch / Map iteration) successfully. + +```json +"Done": { + "Type": "Succeed", + "Output": { + "status": "completed", + "processedAt": "{% $now() %}" + } +} +``` + +Without `Output`, passes input through as output. No `Next` field allowed. + +--- + +## Fail State + +Terminates the state machine with an error. + +```json +"OrderFailed": { + "Type": "Fail", + "Error": "OrderValidationError", + "Cause": "The order could not be validated" +} +``` + +### Dynamic Error and Cause + +```json +"DynamicFail": { + "Type": "Fail", + "Error": "{% $states.input.errorCode %}", + "Cause": "{% $states.input.errorMessage %}" +} +``` + +Build rich, defensive error messages with fallbacks for missing fields: + +```json +"OrderProcessingFailed": { + "Type": "Fail", + "Error": "OrderProcessingError", + "Cause": "{% 'Failed to process order ' & ($exists($orderId) ? $orderId : 'unknown') & ': ' & ($exists($error.Error) ? $error.Error : 'Unknown error') & ' - ' & ($exists($error.Cause) ? $error.Cause : 'No details available') & '. Timestamp: ' & $now() %}" +} +``` + +No `Next`, `End`, `Output`, or `Assign` fields. Fail states are always terminal. + +--- + +## Parallel State + +Executes multiple branches concurrently. All branches receive the same input. + +```json +"LookupCustomerInfo": { + "Type": "Parallel", + "Arguments": { + "customerId": "{% $states.input.customerId %}" + }, + "Branches": [ + { + "StartAt": "GetAddress", + "States": { + "GetAddress": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:GetAddress:$LATEST", + "Payload": "{% $states.input %}" + }, + "Output": "{% $states.result.Payload %}", + "End": true + } + } + }, + { + "StartAt": "GetOrders", + "States": { + "GetOrders": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:GetOrders:$LATEST", + "Payload": "{% $states.input %}" + }, + "Output": "{% $states.result.Payload %}", + "End": true + } + } + } + ], + "Assign": { + "address": "{% $states.result[0] %}", + "orders": "{% $states.result[1] %}" + }, + "Output": { + "address": "{% $states.result[0] %}", + "orders": "{% $states.result[1] %}" + }, + "Next": "ProcessResults" +} +``` + +Key points: +- `Arguments` provides input to each branch's StartAt state (optional, defaults to state input). +- Result is an array with one element per branch, in the same order as `Branches`. +- If any branch fails, the entire Parallel state fails (unless caught). +- States inside branches can only transition to other states within the same branch. +- Branch variables are scoped — branches cannot access each other's variables. +- Use `Output` on terminal states within branches to pass data back to the outer scope. + +--- + +## Map State + +Iterates over an array, processing each element (potentially in parallel). + +### Basic Map + +```json +"ProcessItems": { + "Type": "Map", + "Items": "{% $states.input.orders %}", + "MaxConcurrency": 10, + "ItemProcessor": { + "StartAt": "ProcessOrder", + "States": { + "ProcessOrder": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:ProcessOrder:$LATEST", + "Payload": "{% $states.input %}" + }, + "Output": "{% $states.result.Payload %}", + "End": true + } + } + }, + "Output": "{% $states.result %}", + "Next": "AllDone" +} +``` + +### Map with ItemSelector + +Use `ItemSelector` to reshape each item before processing: + +```json +"ProcessItems": { + "Type": "Map", + "Items": "{% $states.input.detail.shipped %}", + "ItemSelector": { + "parcel": "{% $states.context.Map.Item.Value %}", + "index": "{% $states.context.Map.Item.Index %}", + "courier": "{% $states.input.detail.delivery-partner %}" + }, + "MaxConcurrency": 0, + "ItemProcessor": { + "StartAt": "Ship", + "States": { + "Ship": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:ShipItem:$LATEST", + "Payload": "{% $states.input %}" + }, + "Output": "{% $states.result.Payload %}", + "End": true + } + } + }, + "Next": "Done" +} +``` + +### Map Context Variables + +Inside `ItemSelector`, you can access: +- `$states.context.Map.Item.Value` — the current array element +- `$states.context.Map.Item.Index` — the zero-based index + +### Key Map Fields + +| Field | Description | +|-------|-------------| +| `Items` | JSON array or JSONata expression evaluating to an array | +| `ItemProcessor` | State machine to run for each item (has `StartAt` and `States`) | +| `ItemSelector` | Reshape each item before processing | +| `MaxConcurrency` | Max parallel iterations (0 = unlimited, 1 = sequential) | +| `ToleratedFailurePercentage` | 0-100, percentage of items allowed to fail | +| `ToleratedFailureCount` | Number of items allowed to fail | +| `ItemReader` | Read items from an external resource | +| `ItemBatcher` | Batch items into sub-arrays | +| `ResultWriter` | Write results to an external resource | + +### ProcessorConfig + +The `ItemProcessor` can include a `ProcessorConfig` to control execution mode: + +```json +"ItemProcessor": { + "ProcessorConfig": { + "Mode": "INLINE" + }, + "StartAt": "ProcessOrder", + "States": { ... } +} +``` + +- `INLINE` (default) — iterations run within the parent execution. Use for most cases. +- `DISTRIBUTED` — iterations run as child executions. Use for large-scale processing (thousands+ items), items read from S3, or when you need per-iteration execution history. + +### Failure Tolerance + +```json +"ProcessWithTolerance": { + "Type": "Map", + "Items": "{% $states.input.records %}", + "ToleratedFailurePercentage": 10, + "ToleratedFailureCount": 5, + "ItemProcessor": { ... }, + "Next": "Done" +} +``` + +The Map state fails if either threshold is breached. + +--- \ No newline at end of file diff --git a/aws-step-functions-jsonata/steering/error-handling.md b/aws-step-functions-jsonata/steering/error-handling.md new file mode 100644 index 0000000..b27ad18 --- /dev/null +++ b/aws-step-functions-jsonata/steering/error-handling.md @@ -0,0 +1,445 @@ +# Error Handling in JSONata Mode + +## Overview + +When a state encounters an error, Step Functions defaults to failing the entire execution. You can override this with `Retry` (retry the failed state) and `Catch` (transition to a fallback state). + +`Retry` and `Catch` are available on: Task, Parallel, and Map states. + +## Error Names + +Errors are identified by case-sensitive strings. Step Functions defines these built-in error codes: + +| Error Code | Description | +|-----------|-------------| +| `States.ALL` | Wildcard — matches any error | +| `States.Timeout` | Task exceeded `TimeoutSeconds` or missed heartbeat | +| `States.HeartbeatTimeout` | Task missed heartbeat interval | +| `States.TaskFailed` | Task failed during execution | +| `States.Permissions` | Insufficient privileges | +| `States.ResultPathMatchFailure` | ResultPath cannot be applied (JSONPath only) | +| `States.ParameterPathFailure` | Parameter path resolution failed (JSONPath only) | +| `States.QueryEvaluationError` | JSONata expression evaluation failed | +| `States.BranchFailed` | A Parallel state branch failed | +| `States.NoChoiceMatched` | No Choice rule matched and no Default | +| `States.IntrinsicFailure` | Intrinsic function failed (JSONPath only) | +| `States.ExceedToleratedFailureThreshold` | Map state exceeded failure tolerance | +| `States.ItemReaderFailed` | Map state ItemReader failed | +| `States.ResultWriterFailed` | Map state ResultWriter failed | + +Custom error names are allowed but must NOT start with `States.`. + +--- + +## Retry + +The `Retry` field is an array of Retrier objects. The interpreter scans retriers in order and uses the first one whose `ErrorEquals` matches. + +### Retrier Fields + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `ErrorEquals` | string[] | Required | Error names to match | +| `IntervalSeconds` | integer | 1 | Seconds before first retry | +| `MaxAttempts` | integer | 3 | Maximum retry attempts (0 = never retry) | +| `BackoffRate` | number | 2.0 | Multiplier for retry interval (must be ≥ 1.0) | +| `MaxDelaySeconds` | integer | — | Cap on retry interval | +| `JitterStrategy` | string | — | Jitter strategy (e.g., `"FULL"`) | + +### Basic Retry + +```json +"ProcessPayment": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:Pay:$LATEST", + "Payload": "{% $states.input %}" + }, + "Retry": [ + { + "ErrorEquals": ["States.TaskFailed"], + "IntervalSeconds": 2, + "MaxAttempts": 3, + "BackoffRate": 2.0 + } + ], + "Next": "Confirm" +} +``` + +This retries after 2s, 4s, 8s (3 attempts with 2x backoff). + +### Retry with Max Delay and Jitter + +```json +"Retry": [ + { + "ErrorEquals": ["States.TaskFailed"], + "IntervalSeconds": 1, + "MaxAttempts": 5, + "BackoffRate": 2.0, + "MaxDelaySeconds": 30, + "JitterStrategy": "FULL" + } +] +``` + +### Multiple Retriers + +Retriers are evaluated in order. Each retrier tracks its own attempt count independently: + +```json +"Retry": [ + { + "ErrorEquals": ["ThrottlingException"], + "IntervalSeconds": 1, + "MaxAttempts": 5, + "BackoffRate": 2.0, + "JitterStrategy": "FULL" + }, + { + "ErrorEquals": ["States.Timeout"], + "MaxAttempts": 0 + }, + { + "ErrorEquals": ["States.ALL"], + "IntervalSeconds": 3, + "MaxAttempts": 2, + "BackoffRate": 1.5 + } +] +``` + +Rules: +- `States.ALL` must appear alone in its `ErrorEquals` array. +- `States.ALL` must be in the last retrier. +- `MaxAttempts: 0` means "never retry this error." +- Retrier attempt counts reset when the interpreter transitions to another state. + +--- + +## Catch + +The `Catch` field is an array of Catcher objects. After retries are exhausted (or if no retrier matches), the interpreter scans catchers in order. + +### Catcher Fields (JSONata) + +| Field | Type | Description | +|-------|------|-------------| +| `ErrorEquals` | string[] | Required. Error names to match | +| `Next` | string | Required. State to transition to | +| `Output` | any | Optional. Transform the error output | +| `Assign` | object | Optional. Assign variables from error context | + +### Basic Catch + +```json +"ProcessOrder": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:Process:$LATEST", + "Payload": "{% $states.input %}" + }, + "Catch": [ + { + "ErrorEquals": ["ValidationError"], + "Output": { + "error": "{% $states.errorOutput.Error %}", + "cause": "{% $states.errorOutput.Cause %}", + "originalInput": "{% $states.input %}" + }, + "Next": "HandleValidationError" + }, + { + "ErrorEquals": ["States.ALL"], + "Output": "{% $states.errorOutput %}", + "Next": "HandleGenericError" + } + ], + "Next": "Success" +} +``` + +### Error Output Structure + +When a state fails and matches a Catcher, the Error Output is a JSON object with: +- `Error` (string) — the error name +- `Cause` (string) — human-readable error description + +```json +{ + "Error": "States.TaskFailed", + "Cause": "Lambda function returned an error" +} +``` + +### Catch with Variable Assignment + +```json +"Catch": [ + { + "ErrorEquals": ["States.ALL"], + "Assign": { + "hasError": true, + "errorType": "{% $states.errorOutput.Error %}", + "errorMessage": "{% $states.errorOutput.Cause %}" + }, + "Output": "{% $merge([$states.input, {'error': $states.errorOutput}]) %}", + "Next": "ErrorHandler" + } +] +``` + +In a Catch block, `Assign` and `Output` can reference: +- `$states.input` — the original state input +- `$states.errorOutput` — the error details +- `$states.context` — execution context + +If a Catcher matches, the state's top-level `Assign` is NOT evaluated — only the Catcher's `Assign` runs. + +### Catch Without Output + +If no `Output` is provided in the Catcher, the state output is the raw Error Output object. + +### Building Rich Error Context for Fail States + +A user-friendly pattern is to capture error details into a variable via Catch `Assign`, then reference that variable in a Fail state's `Cause` with defensive fallbacks: + +```json +"ChargePayment": { + "Type": "Task", + "Resource": "arn:aws:states:::sqs:sendMessage", + "Arguments": { ... }, + "Catch": [ + { + "ErrorEquals": ["States.ALL"], + "Assign": { + "error": "{% $states.errorOutput %}" + }, + "Next": "PaymentFailed" + } + ], + "Next": "ConfirmOrder" +}, +"PaymentFailed": { + "Type": "Fail", + "Error": "PaymentError", + "Cause": "{% 'Payment failed for order ' & ($exists($orderId) ? $orderId : 'unknown') & ': ' & ($exists($error.Error) ? $error.Error : 'Unknown') & ' - ' & ($exists($error.Cause) ? $error.Cause : 'No details') & '. Timestamp: ' & $now() %}" +} +``` + +Always guard with `$exists()` — if the variable was never assigned (e.g., the Catch didn't fire for that path), referencing it directly throws `States.QueryEvaluationError`. + +--- + +## Combined Retry and Catch + +When both are present, retries are attempted first. Only if retries are exhausted does the Catch apply: + +```json +"CallExternalAPI": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:CallAPI:$LATEST", + "Payload": "{% $states.input %}" + }, + "Retry": [ + { + "ErrorEquals": ["ThrottlingException", "ServiceUnavailable"], + "IntervalSeconds": 2, + "MaxAttempts": 3, + "BackoffRate": 2.0, + "JitterStrategy": "FULL" + }, + { + "ErrorEquals": ["States.Timeout"], + "IntervalSeconds": 5, + "MaxAttempts": 2 + } + ], + "Catch": [ + { + "ErrorEquals": ["ThrottlingException", "ServiceUnavailable"], + "Assign": { + "retryExhausted": true + }, + "Output": { + "error": "Service temporarily unavailable after retries", + "details": "{% $states.errorOutput %}" + }, + "Next": "NotifyAndRetryLater" + }, + { + "ErrorEquals": ["States.ALL"], + "Output": { + "error": "{% $states.errorOutput %}", + "input": "{% $states.input %}" + }, + "Next": "FatalErrorHandler" + } + ], + "Output": "{% $states.result.Payload %}", + "Next": "ProcessResponse" +} +``` + +--- + +## Handling States.QueryEvaluationError + +JSONata expressions can fail at runtime. Common causes: + +1. **Type error**: `{% $x + $y %}` where `$x` or `$y` is not a number +2. **Type incompatibility**: `"TimeoutSeconds": "{% $name %}"` where `$name` is a string +3. **Value out of range**: Negative number for `TimeoutSeconds` +4. **Undefined result**: `{% $data.nonExistentField %}` — JSON cannot represent undefined + +All of these throw `States.QueryEvaluationError`. Handle it like any other error: + +```json +"Retry": [ + { + "ErrorEquals": ["States.QueryEvaluationError"], + "MaxAttempts": 0 + } +], +"Catch": [ + { + "ErrorEquals": ["States.QueryEvaluationError"], + "Output": { + "error": "Data transformation failed", + "details": "{% $states.errorOutput %}" + }, + "Next": "HandleDataError" + } +] +``` + +### Preventing QueryEvaluationError + +Use defensive JSONata expressions: + +```json +"Output": { + "name": "{% $exists($states.input.name) ? $states.input.name : 'Unknown' %}", + "total": "{% $type($states.input.amount) = 'number' ? $states.input.amount : 0 %}" +} +``` + +Watch out for single-value vs array results from filters. JSONata returns a single object (not a 1-element array) when a filter matches exactly one item, and undefined when nothing matches. Both cases will throw `States.QueryEvaluationError` if you pass the result to array-expecting functions like `$count`, `$map`, or a Map state `Items` field. + +Guard filtered results before using them: + +```json +"Assign": { + "pendingOrders": "{% ($filtered := $states.input.orders[status = 'pending']; $type($filtered) = 'array' ? $filtered : $exists($filtered) ? [$filtered] : []) %}" +} +``` + +This ensures `$pendingOrders` is always an array regardless of how many items matched. + +--- + +## Error Handling in Parallel States + +If any branch fails, the entire Parallel state fails. Catch the error at the Parallel state level: + +```json +"ParallelWork": { + "Type": "Parallel", + "Branches": [ ... ], + "Retry": [ + { + "ErrorEquals": ["States.BranchFailed"], + "MaxAttempts": 1 + } + ], + "Catch": [ + { + "ErrorEquals": ["States.ALL"], + "Output": { + "error": "{% $states.errorOutput %}", + "failedAt": "parallel execution" + }, + "Next": "HandleParallelError" + } + ], + "Next": "Continue" +} +``` + +--- + +## Error Handling in Map States + +Individual iteration failures can be tolerated: + +```json +"ProcessAll": { + "Type": "Map", + "Items": "{% $states.input.records %}", + "ToleratedFailurePercentage": 10, + "ItemProcessor": { ... }, + "Catch": [ + { + "ErrorEquals": ["States.ExceedToleratedFailureThreshold"], + "Output": { + "error": "Too many items failed", + "details": "{% $states.errorOutput %}" + }, + "Next": "HandleBatchFailure" + }, + { + "ErrorEquals": ["States.ALL"], + "Next": "HandleMapError" + } + ], + "Next": "Done" +} +``` + +--- + +## Common Error Handling Patterns + +### Circuit Breaker with Variables + +```json +"CheckRetryCount": { + "Type": "Choice", + "Choices": [ + { + "Condition": "{% $retryCount >= $maxRetries %}", + "Next": "MaxRetriesExceeded" + } + ], + "Default": "AttemptOperation" +}, +"AttemptOperation": { + "Type": "Task", + "Resource": "...", + "Assign": { + "retryCount": "{% $retryCount + 1 %}" + }, + "Catch": [ + { + "ErrorEquals": ["States.ALL"], + "Assign": { + "retryCount": "{% $retryCount + 1 %}", + "lastError": "{% $states.errorOutput %}" + }, + "Next": "WaitBeforeRetry" + } + ], + "Next": "Success" +}, +"WaitBeforeRetry": { + "Type": "Wait", + "Seconds": "{% $power(2, $retryCount) %}", + "Next": "CheckRetryCount" +} +``` + diff --git a/aws-step-functions-jsonata/steering/service-integrations.md b/aws-step-functions-jsonata/steering/service-integrations.md new file mode 100644 index 0000000..490c104 --- /dev/null +++ b/aws-step-functions-jsonata/steering/service-integrations.md @@ -0,0 +1,485 @@ +# Service Integrations in JSONata Mode + +## Integration Types + +Step Functions can integrate with AWS services in three patterns: + +1. **Optimized integrations** — Purpose-built, recommended where available (e.g., Lambda, DynamoDB, SNS, SQS, ECS, Glue, SageMaker, etc.) +2. **AWS SDK integrations** — Call any AWS SDK API action directly +3. **HTTP Task** — Call HTTPS APIs (e.g., Stripe, Salesforce) + +### Resource ARN Patterns + +``` +# Optimized integration +"Resource": "arn:aws:states:::servicename:apiAction" + +# Optimized integration (synchronous — wait for completion) +"Resource": "arn:aws:states:::servicename:apiAction.sync" + +# Optimized integration (wait for callback token) +"Resource": "arn:aws:states:::servicename:apiAction.waitForTaskToken" + +# AWS SDK integration +"Resource": "arn:aws:states:::aws-sdk:serviceName:apiAction" +``` + +--- + +## Lambda Function + +### Optimized Integration (Recommended) + +```json +"InvokeFunction": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:MyFunction:$LATEST", + "Payload": { + "orderId": "{% $states.input.orderId %}", + "customer": "{% $states.input.customer %}" + } + }, + "Output": "{% $states.result.Payload %}", + "Next": "NextState" +} +``` + +Always include a version qualifier (`:$LATEST`, `:1`, or an alias like `:prod`) on the function ARN. + +The result is wrapped in a `Payload` field, so use `$states.result.Payload` to access the Lambda return value. + +### SDK Integration + +```json +"InvokeViaSDK": { + "Type": "Task", + "Resource": "arn:aws:states:::aws-sdk:lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:MyFunction", + "Payload": "{% $string($states.input) %}" + }, + "Next": "NextState" +} +``` + +--- + +## DynamoDB + +### GetItem + +```json +"GetUser": { + "Type": "Task", + "Resource": "arn:aws:states:::dynamodb:getItem", + "Arguments": { + "TableName": "UsersTable", + "Key": { + "userId": { + "S": "{% $states.input.userId %}" + } + } + }, + "Assign": { + "user": "{% $states.result.Item %}" + }, + "Output": "{% $states.result.Item %}", + "Next": "ProcessUser" +} +``` + +### PutItem + +```json +"SaveOrder": { + "Type": "Task", + "Resource": "arn:aws:states:::dynamodb:putItem", + "Arguments": { + "TableName": "OrdersTable", + "Item": { + "orderId": { + "S": "{% $orderId %}" + }, + "status": { + "S": "processing" + }, + "total": { + "N": "{% $string($states.input.total) %}" + }, + "createdAt": { + "S": "{% $now() %}" + } + } + }, + "Next": "ProcessOrder" +} +``` + +### UpdateItem + +```json +"UpdateStatus": { + "Type": "Task", + "Resource": "arn:aws:states:::dynamodb:updateItem", + "Arguments": { + "TableName": "OrdersTable", + "Key": { + "orderId": { + "S": "{% $orderId %}" + } + }, + "UpdateExpression": "SET #s = :status, updatedAt = :time", + "ExpressionAttributeNames": { + "#s": "status" + }, + "ExpressionAttributeValues": { + ":status": { + "S": "{% $states.input.newStatus %}" + }, + ":time": { + "S": "{% $now() %}" + } + } + }, + "Next": "Done" +} +``` + +### Query + +```json +"QueryOrders": { + "Type": "Task", + "Resource": "arn:aws:states:::aws-sdk:dynamodb:query", + "Arguments": { + "TableName": "OrdersTable", + "KeyConditionExpression": "customerId = :cid", + "ExpressionAttributeValues": { + ":cid": { + "S": "{% $states.input.customerId %}" + } + } + }, + "Output": "{% $states.result.Items %}", + "Next": "ProcessOrders" +} +``` + +--- + +## SNS (Simple Notification Service) + +### Publish Message + +```json +"SendNotification": { + "Type": "Task", + "Resource": "arn:aws:states:::sns:publish", + "Arguments": { + "TopicArn": "arn:aws:sns:us-east-1:123456789012:OrderNotifications", + "Message": "{% 'Order ' & $orderId & ' has been processed successfully.' %}", + "Subject": "Order Confirmation" + }, + "Next": "Done" +} +``` + +### Publish with JSON Message + +```json +"SendStructuredNotification": { + "Type": "Task", + "Resource": "arn:aws:states:::sns:publish", + "Arguments": { + "TopicArn": "arn:aws:sns:us-east-1:123456789012:Alerts", + "Message": "{% $string({'orderId': $orderId, 'status': $states.input.status, 'timestamp': $now()}) %}" + }, + "Next": "Done" +} +``` + +--- + +## SQS (Simple Queue Service) + +### Send Message + +```json +"QueueMessage": { + "Type": "Task", + "Resource": "arn:aws:states:::sqs:sendMessage", + "Arguments": { + "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/ProcessingQueue", + "MessageBody": "{% $string($states.input) %}" + }, + "Next": "Done" +} +``` + +### Send Message with Wait for Task Token + +```json +"WaitForApproval": { + "Type": "Task", + "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken", + "Arguments": { + "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/ApprovalQueue", + "MessageBody": "{% $string({'taskToken': $states.context.Task.Token, 'orderId': $orderId, 'amount': $states.input.amount}) %}" + }, + "TimeoutSeconds": 86400, + "Next": "ProcessApproval" +} +``` + +The execution pauses until an external system calls `SendTaskSuccess` or `SendTaskFailure` with the task token. + +--- + +## Step Functions (Nested Execution) + +### Start Execution (Synchronous) + +```json +"RunSubWorkflow": { + "Type": "Task", + "Resource": "arn:aws:states:::states:startExecution.sync:2", + "Arguments": { + "StateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:ChildWorkflow", + "Input": "{% $states.input %}" + }, + "Output": "{% $parse($states.result.Output) %}", + "Next": "ProcessSubResult" +} +``` + +Note: The `.sync:2` suffix waits for completion. The child output is a JSON string in `$states.result.Output`, so use `$parse()` to deserialize it. + +### Start Execution (Async — Fire and Forget) + +```json +"StartAsync": { + "Type": "Task", + "Resource": "arn:aws:states:::states:startExecution", + "Arguments": { + "StateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:AsyncWorkflow", + "Input": "{% $string($states.input) %}" + }, + "Next": "Continue" +} +``` + +--- + +## EventBridge + +### Put Events + +```json +"EmitEvent": { + "Type": "Task", + "Resource": "arn:aws:states:::events:putEvents", + "Arguments": { + "Entries": [ + { + "Source": "my.application", + "DetailType": "OrderProcessed", + "Detail": "{% $string({'orderId': $orderId, 'status': 'completed'}) %}", + "EventBusName": "default" + } + ] + }, + "Next": "Done" +} +``` + +--- + +## ECS / Fargate + +### Run Task (Synchronous) + +```json +"RunContainer": { + "Type": "Task", + "Resource": "arn:aws:states:::ecs:runTask.sync", + "Arguments": { + "LaunchType": "FARGATE", + "Cluster": "arn:aws:ecs:us-east-1:123456789012:cluster/MyCluster", + "TaskDefinition": "arn:aws:ecs:us-east-1:123456789012:task-definition/MyTask:1", + "NetworkConfiguration": { + "AwsvpcConfiguration": { + "Subnets": ["subnet-abc123"], + "SecurityGroups": ["sg-abc123"], + "AssignPublicIp": "ENABLED" + } + }, + "Overrides": { + "ContainerOverrides": [ + { + "Name": "my-container", + "Environment": [ + { + "Name": "ORDER_ID", + "Value": "{% $orderId %}" + } + ] + } + ] + } + }, + "TimeoutSeconds": 600, + "Next": "Done" +} +``` + +--- + +## AWS Glue + +### Start Job Run (Synchronous) + +```json +"RunGlueJob": { + "Type": "Task", + "Resource": "arn:aws:states:::glue:startJobRun.sync", + "Arguments": { + "JobName": "my-etl-job", + "Arguments": { + "--input_path": "{% $states.input.inputPath %}", + "--output_path": "{% $states.input.outputPath %}" + } + }, + "TimeoutSeconds": 3600, + "Next": "Done" +} +``` + +--- + +## Amazon Bedrock + +### Invoke Model + +```json +"InvokeModel": { + "Type": "Task", + "Resource": "arn:aws:states:::bedrock:invokeModel", + "Arguments": { + "ModelId": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0", + "ContentType": "application/json", + "Accept": "application/json", + "Body": { + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 1024, + "messages": [ + { + "role": "user", + "content": "{% $states.input.prompt %}" + } + ] + } + }, + "Output": "{% $states.result.Body %}", + "Next": "ProcessResponse" +} +``` + +--- + +## S3 + +### GetObject + +```json +"ReadFile": { + "Type": "Task", + "Resource": "arn:aws:states:::aws-sdk:s3:getObject", + "Arguments": { + "Bucket": "my-bucket", + "Key": "{% $states.input.filePath %}" + }, + "Output": "{% $states.result.Body %}", + "Next": "ProcessFile" +} +``` + +### PutObject + +```json +"WriteFile": { + "Type": "Task", + "Resource": "arn:aws:states:::aws-sdk:s3:putObject", + "Arguments": { + "Bucket": "my-bucket", + "Key": "{% 'results/' & $orderId & '.json' %}", + "Body": "{% $string($states.input.results) %}" + }, + "Next": "Done" +} +``` + +--- + +## Cross-Account Access + +Use the `Credentials` field to assume a role in another account: + +```json +"CrossAccountCall": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Credentials": { + "RoleArn": "arn:aws:iam::111122223333:role/CrossAccountRole" + }, + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:111122223333:function:RemoteFunction:$LATEST", + "Payload": "{% $states.input %}" + }, + "Output": "{% $states.result.Payload %}", + "Next": "Done" +} +``` + +--- + +## Synchronous vs Asynchronous Patterns + +| Pattern | Resource Suffix | Behavior | +|---------|----------------|----------| +| Request-Response | (none) | Call API and continue immediately | +| Synchronous | `.sync` | Wait for task to complete | +| Wait for Callback | `.waitForTaskToken` | Pause until external callback | + +### When to Use Each + +- **Request-Response**: Fire-and-forget operations (start a process, send a message) +- **Synchronous (`.sync`)**: When you need the result before continuing (run ECS task, execute child workflow, run Glue job) +- **Wait for Callback (`.waitForTaskToken`)**: Human approval, external system processing, long-running async operations + +### Callback Pattern Example + +```json +"WaitForHumanApproval": { + "Type": "Task", + "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken", + "Arguments": { + "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/ApprovalQueue", + "MessageBody": "{% $string({'taskToken': $states.context.Task.Token, 'request': $states.input}) %}" + }, + "TimeoutSeconds": 604800, + "Catch": [ + { + "ErrorEquals": ["States.Timeout"], + "Output": { + "status": "approval_timeout" + }, + "Next": "HandleTimeout" + } + ], + "Next": "ApprovalReceived" +} +``` + +The external system must call `SendTaskSuccess` or `SendTaskFailure` with the task token to resume execution. diff --git a/aws-step-functions-jsonata/steering/variables-and-data.md b/aws-step-functions-jsonata/steering/variables-and-data.md new file mode 100644 index 0000000..8e4339b --- /dev/null +++ b/aws-step-functions-jsonata/steering/variables-and-data.md @@ -0,0 +1,498 @@ +# Variables and Data Transformation (JSONata Mode) + +## JSONata Expression Syntax + +JSONata expressions are written inside `{% %}` delimiters in string values: + +```json +"Output": "{% $states.input.customer.name %}" +"TimeoutSeconds": "{% $timeout %}" +"Condition": "{% $states.input.age >= 18 %}" +``` + +Rules: +- The string must start with `{%` (no leading spaces) and end with `%}` (no trailing spaces). +- Not all fields accept JSONata — `Type` and `Resource` must be constant strings. +- JSONata expressions can appear in string values within objects and arrays at any nesting depth. +- A string without `{% %}` is treated as a literal value. +- All string literals inside JSONata expressions must use single quotes (`'text'`), not double quotes. The expression is already inside a JSON double-quoted string, so double quotes would break the JSON. +- Use `:=` inside `( ... )` blocks to bind local variables within a single expression. These are expression-local only — they do NOT set state machine variables (use `Assign` for that). +- Complex logic is wrapped in `( expr1; expr2; ...; finalExpr )` where semicolons separate sequential expressions and the last expression is the return value. + +### String Quoting + +```json +"Output": "{% 'Hello ' & $states.input.name %}" +"Condition": "{% $states.input.status = 'active' %}" +``` + +Never use double quotes inside the expression: +``` +❌ "Output": "{% "Hello" %}" +✓ "Output": "{% 'Hello' %}" +``` + +### Local Variable Binding with `:=` + +Use `:=` inside `( ... )` blocks to bind intermediate values within a single JSONata expression. Semicolons separate each binding, and the last expression is the return value: + +```json +"Output": "{% ( $subtotal := $sum($states.input.items.price); $tax := $subtotal * 0.1; $discount := $exists($couponValue) ? $couponValue : 0; {'subtotal': $subtotal, 'tax': $tax, 'discount': $discount, 'total': $subtotal + $tax - $discount} ) %}" +``` + +You can also define local helper functions: + +```json +"Assign": { + "summary": "{% ( $formatPrice := function($amt) { '$' & $formatNumber($amt, '#,##0.00') }; $subtotal := $sum($states.input.items.price); {'itemCount': $count($states.input.items), 'subtotal': $formatPrice($subtotal), 'total': $formatPrice($subtotal * 1.1)} ) %}" +} +``` + +Local variables bound with `:=` exist only within the `( ... )` block. They do not affect state machine variables. To persist values across states, use the `Assign` field. + +## The `$states` Reserved Variable + +Step Functions provides a reserved `$states` variable in every JSONata state: + +``` +$states = { + "input": // Original input to the state + "result": // Task/Parallel/Map result (if successful) + "errorOutput": // Error Output (only available in Catch) + "context": // Context object (execution metadata) +} +``` + +### Where Each Field Is Accessible + +| Field | Accessible In | +|-------|--------------| +| `$states.input` | All fields that accept JSONata, in any state | +| `$states.result` | Top-level `Output` and `Assign` in Task, Parallel, Map states | +| `$states.errorOutput` | `Output` and `Assign` inside a `Catch` block | +| `$states.context` | All fields that accept JSONata, in any state | + +### Context Object + +`$states.context` provides execution metadata: + +```json +"executionId": "{% $states.context.Execution.Id %}", +"startTime": "{% $states.context.Execution.StartTime %}", +"stateName": "{% $states.context.State.Name %}", +"originalInput": "{% $states.context.Execution.Input %}" +``` + +Useful context fields: +- `$states.context.Execution.Id` — Execution ARN +- `$states.context.Execution.Input` — Original workflow input +- `$states.context.Execution.Name` — Execution name +- `$states.context.Execution.StartTime` — When execution started +- `$states.context.State.Name` — Current state name +- `$states.context.State.EnteredTime` — When current state was entered +- `$states.context.StateMachine.Id` — State machine ARN +- `$states.context.StateMachine.Name` — State machine name + +Inside Map state `ItemSelector`: +- `$states.context.Map.Item.Value` — Current array element +- `$states.context.Map.Item.Index` — Zero-based index + +## JSONata Restrictions in Step Functions + +1. **No `$` or `$$` at top level**: You cannot use `$` or `$$` to reference an implicit input document. Use `$states.input` instead. + - Invalid: `"Output": "{% $.name %}"` (top-level `$`) + - Valid: `"Output": "{% $states.input.name %}"` + - Valid inside expressions: `"Output": "{% $states.input.items[$.price > 10] %}"` (nested `$` is OK) + +2. **No unqualified field names at top level**: Use variables or `$states.input`. + - Invalid: `"Output": "{% name %}"` (unqualified) + - Valid: `"Output": "{% $states.input.name %}"` + +3. **No `$eval`**: Use `$parse()` instead for deserializing JSON strings. + +4. **Expressions must produce a defined value**: `$data.nonExistentField` throws `States.QueryEvaluationError` because JSON cannot represent undefined. + +--- + +## Workflow Variables with `Assign` + +Variables let you store data in one state and reference it in any subsequent state, without threading data through Output/Input chains. + +### Declaring Variables + +```json +"StoreData": { + "Type": "Pass", + "Assign": { + "productName": "product1", + "count": 42, + "available": true, + "config": "{% $states.input.configuration %}" + }, + "Next": "UseData" +} +``` + +### Referencing Variables + +Prepend the variable name with `$`: + +```json +"Arguments": { + "product": "{% $productName %}", + "quantity": "{% $count %}" +} +``` + +### Assigning from Task Results + +```json +"FetchPrice": { + "Type": "Task", + "Resource": "arn:aws:states:::lambda:invoke", + "Arguments": { + "FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:GetPrice:$LATEST", + "Payload": { + "product": "{% $states.input.product %}" + } + }, + "Assign": { + "currentPrice": "{% $states.result.Payload.price %}" + }, + "Output": "{% $states.result.Payload %}", + "Next": "CheckPrice" +} +``` + +### States That Support Assign + +Pass, Task, Map, Parallel, Choice, Wait — all support `Assign`. + +Succeed and Fail do NOT support `Assign`. + +### Assign in Choice Rules and Catch + +Choice Rules and Catch blocks can each have their own `Assign`: + +```json +"CheckValue": { + "Type": "Choice", + "Choices": [ + { + "Condition": "{% $states.input.value > 100 %}", + "Assign": { + "tier": "premium" + }, + "Next": "PremiumPath" + } + ], + "Default": "StandardPath", + "Assign": { + "tier": "standard" + } +} +``` + +If a Choice Rule matches, its `Assign` is used. If no rule matches, the state-level `Assign` is used. + +--- + +## Variable Evaluation Order + +All expressions in `Assign` are evaluated using variable values as they were on state entry. New values only take effect in the next state. + +```json +"SwapExample": { + "Type": "Pass", + "Assign": { + "x": "{% $y %}", + "y": "{% $x %}" + }, + "Next": "AfterSwap" +} +``` + +If `$x = 3` and `$y = 6` on entry, after this state: `$x = 6`, `$y = 3`. This works because all expressions are evaluated first, then assignments are made. + +You cannot assign to a sub-path of a variable: +- Valid: `"Assign": {"x": 42}` +- Invalid: `"Assign": {"x.y": 42}` or `"Assign": {"x[2]": 42}` + +--- + +## Variable Scope + +Variables exist in a state-machine-local scope: + +- **Outer scope**: All states in the top-level `States` field. +- **Inner scope**: States inside a Parallel branch or Map iteration. + +### Scope Rules + +1. Inner scopes can READ variables from outer scopes. +2. Inner scopes CANNOT ASSIGN to variables that exist in an outer scope. +3. Variable names must be unique across outer and inner scopes (no shadowing). +4. Variables in different Parallel branches or Map iterations are isolated from each other. +5. When a Parallel branch or Map iteration completes, its variables go out of scope. +6. Exception: Distributed Map states cannot reference variables in outer scopes. + +### Passing Data Out of Inner Scopes + +Use `Output` on terminal states within branches/iterations to return data to the outer scope: + +```json +"ParallelWork": { + "Type": "Parallel", + "Branches": [ + { + "StartAt": "BranchA", + "States": { + "BranchA": { + "Type": "Task", + "Resource": "...", + "Output": "{% $states.result.Payload %}", + "End": true + } + } + } + ], + "Assign": { + "branchAResult": "{% $states.result[0] %}" + }, + "Next": "Continue" +} +``` + +### Catch Assign and Outer Scope + +In a Catch block on a Parallel or Map state, `Assign` can assign values to variables in the outer scope (the scope where the Parallel/Map state exists): + +```json +"Catch": [ + { + "ErrorEquals": ["States.ALL"], + "Assign": { + "errorOccurred": true, + "errorDetails": "{% $states.errorOutput %}" + }, + "Next": "HandleError" + } +] +``` + +--- + +## Arguments and Output Fields + +### Arguments + +Provides input to Task and Parallel states (replaces JSONPath `Parameters`): + +```json +"Arguments": { + "staticField": "hello", + "dynamicField": "{% $states.input.name %}", + "computed": "{% $count($states.input.items) %}" +} +``` + +Or as a single JSONata expression: + +```json +"Arguments": "{% $states.input.payload %}" +``` + +`Arguments` can reference `$states.input` and `$states.context`, but NOT `$states.result` or `$states.errorOutput`. + +### Output + +Transforms the state output (replaces JSONPath `ResultSelector` + `ResultPath` + `OutputPath`): + +```json +"Output": { + "customerId": "{% $states.input.id %}", + "result": "{% $states.result.Payload %}", + "processedAt": "{% $now() %}" +} +``` + +Or as a single expression or literal value: + +```json +"Output": "{% $states.result.Payload %}" +"Output": 42 +"Output": { "status": "done" } +``` + +If `Output` is not provided: +- Task, Parallel, Map: state output = the result +- All other states: state output = the state input + +### Assign and Output Are Parallel + +`Assign` and `Output` are evaluated in parallel. Variable assignments in `Assign` are NOT available in `Output` of the same state — you must re-derive values in both if needed: + +```json +"Assign": { + "savedPrice": "{% $states.result.Payload.price %}" +}, +"Output": { + "price": "{% $states.result.Payload.price %}" +} +``` + +--- + +## Variable Limits + +| Limit | Value | +|-------|-------| +| Max size of a single variable | 256 KiB | +| Max combined size in a single Assign | 256 KiB | +| Max total stored variables per execution | 10 MiB | +| Max variable name length | 80 Unicode characters | + +--- + +## Data Transformation Patterns + +### Filtering Arrays + +```json +"Output": { + "expensiveItems": "{% $states.input.items[price > 100] %}" +} +``` + +### Aggregation + +```json +"Output": { + "total": "{% $sum($states.input.items.price) %}", + "average": "{% $average($states.input.items.price) %}", + "count": "{% $count($states.input.items) %}" +} +``` + +### String Operations + +```json +"Output": { + "fullName": "{% $states.input.firstName & ' ' & $states.input.lastName %}", + "upper": "{% $uppercase($states.input.name) %}", + "trimmed": "{% $trim($states.input.rawInput) %}" +} +``` + +### Object Merging + +```json +"Output": "{% $merge([$states.input, {'processedAt': $now(), 'status': 'complete'}]) %}" +``` + +### Building Lookup Maps with `$reduce` + +Use `$reduce` to transform an array into a key-value object: + +```json +"Assign": { + "priceByProduct": "{% $reduce($states.input.items, function($acc, $item) { $merge([$acc, {$item.productId: $item.price}]) }, {}) %}" +} +``` + +Given `[{"productId": "A1", "price": 10}, {"productId": "B2", "price": 25}]`, this produces `{"A1": 10, "B2": 25}`. + +### Dynamic Key Access with `$lookup` + +Use `$lookup` to access an object property by a variable key: + +```json +"Output": { + "price": "{% $lookup($priceByProduct, $states.input.productId) %}" +} +``` + +This is essential when you've built a mapping object with `$reduce` and need to retrieve values dynamically. Standard dot notation (`$priceByProduct.someKey`) only works with literal key names. + +### Conditional Values + +```json +"Output": { + "tier": "{% $states.input.total > 1000 ? 'gold' : 'standard' %}", + "discount": "{% $exists($states.input.coupon) ? 0.1 : 0 %}" +} +``` + +### Array Membership with `in` and Concatenation with `$append` + +Test if a value exists in an array with `in`: + +```json +"Condition": "{% $states.input.status in ['pending', 'processing', 'shipped'] %}" +``` + +Concatenate arrays with `$append`: + +```json +"Assign": { + "allIds": "{% $append($states.input.orderIds, $states.input.returnIds) %}" +} +``` + +### Array Mapping + +```json +"Output": { + "names": "{% $states.input.users.(firstName & ' ' & lastName) %}" +} +``` + +### Generating UUIDs and Random Values + +```json +"Assign": { + "requestId": "{% $uuid() %}", + "randomValue": "{% $random() %}" +} +``` + +### Partitioning Arrays + +```json +"Assign": { + "batches": "{% $partition($states.input.items, 10) %}" +} +``` + +### Parsing JSON Strings + +```json +"Assign": { + "parsed": "{% $parse($states.input.jsonString) %}" +} +``` + +### Hashing + +```json +"Assign": { + "hash": "{% $hash($states.input.content, 'SHA-256') %}" +} +``` + +### Timestamp Comparison with `$toMillis` + +JSONata timestamps are strings, so you can't compare them directly with `<` or `>`. Use `$toMillis` to convert to numeric milliseconds: + +```json +"Condition": "{% $toMillis($states.input.orderDate) > $toMillis($states.input.cutoffDate) %}" +``` + +Useful for sorting timestamps, calculating durations, or finding the most recent entry: + +```json +"Assign": { + "ageMinutes": "{% $round(($toMillis($now()) - $toMillis($states.input.createdAt)) / 60000, 2) %}", + "mostRecent": "{% $sort($states.input.timestamps, function($a, $b) { $toMillis($b) - $toMillis($a) })[0] %}" +} +```