Experimentation Steps - Add group.experiment() documentation#1444
Open
AndyInternet wants to merge 1 commit intomainfrom
Open
Experimentation Steps - Add group.experiment() documentation#1444AndyInternet wants to merge 1 commit intomainfrom
AndyInternet wants to merge 1 commit intomainfrom
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
3 tasks
AndyInternet
added a commit
to inngest/inngest-js
that referenced
this pull request
Mar 9, 2026
## Summary
This PR introduces a native experimentation framework to the Inngest
TypeScript SDK, allowing developers to run A/B tests and feature
experiments directly within durable functions with guaranteed
consistency and observability.
## 1. Core Tool: `group.experiment()`
The `group.experiment()` tool is the primary entry point for running
experiments. It coordinates variant selection and execution while
ensuring durability.
- **Durable Selection**: The selection process is wrapped in a memoized
step. This ensures that once a variant is chosen for a specific run, the
same variant is used across all replays, even if the selection logic
involves randomness.
- **Flexible Execution**: Each variant is defined as a callback. The
selected variant's callback is executed at the top level, allowing its
internal `step.*` calls to participate in normal step discovery.
- **Result Handling**: By default, it returns the result of the selected
variant's callback. Using `withVariant: true` allows the caller to
receive both the result and the name of the selected variant.
### Example Usage:
```ts
const result = await group.experiment("checkout-flow", {
variants: {
control: () => step.run("old-flow", () => oldLogic()),
treatment: () => step.run("new-flow", () => newLogic()),
},
select: experiment.weighted({ control: 50, treatment: 50 }),
});
```
## 2. Selection Strategies (`experiment.*`)
The PR provides a set of pre-built strategies for variant selection:
- **`experiment.fixed(variantName)`**: Always selects the specified
variant. Useful for manual overrides or testing.
- **`experiment.weighted(weights)`**: Performs weighted random
selection. It is deterministic per run because it is seeded with the
Inngest `runId`.
- **`experiment.bucket(value, options?)`**: Uses consistent hashing
(SHA-256) to map a value (like a `userId`) to a variant. The same input
value will always result in the same variant for a given set of weights.
- **`experiment.custom(fn)`**: Allows developers to provide their own
selection logic (e.g., fetching a flag from an external provider). The
result is still memoized durably by Inngest.
## 3. Observability and Metadata
The framework is deeply integrated with the Inngest platform for better
visibility:
- **Automatic Metadata**: The selection step automatically carries
`inngest.experiment` metadata, including:
- Experiment name
- Selected variant
- Selection strategy used
- Available variants and their weights
- **Context Propagation**: Using `AsyncLocalStorage`, the experiment
context (experiment ID and variant name) is propagated to all steps
executed within the selected variant's callback. This allows the Inngest
UI to group and attribute these steps to the experiment.
- **Dashboard Warnings**: Includes built-in warnings, such as when
`experiment.bucket()` receives a null/undefined value.
## 4. Reliability and Safety
- **Zero-Step Guard**: To ensure durability, the SDK detects if a
variant callback completes without invoking any Inngest step tools.
Since non-step logic re-runs on every replay, the SDK throws a
`NonRetriableError` to force developers to wrap their variant logic in
`step.run()`.
- **Validation**:
- Ensures at least one variant is defined.
- Validates that weights are non-negative and have a positive total.
- Verifies that `custom` or `bucket` strategies return a valid variant
name defined in the experiment.
- **Runtime Support**: Requires `AsyncLocalStorage` for context
propagation and zero-step detection, falling back gracefully where
possible in restricted environments.
## Checklist
<!-- Tick these items off as you progress. -->
<!-- If an item isn't applicable, ideally please strikeout the item by
wrapping it in "~~"" and suffix it with "N/A My reason for skipping
this." -->
<!-- e.g. "- [ ] ~~Added tests~~ N/A Only touches docs" -->
- [x] Added a [docs PR](https://github.com/inngest/website) that
references this PR - [docs
here](inngest/website#1444)
- [x] Added unit/integration tests
- [x] Added changesets if applicable
## Related
<!-- A space for any related links, issues, or PRs. -->
<!-- Linear issues are autolinked. -->
<!-- e.g. - INN-123 -->
<!-- GitHub issues/PRs can be linked using shorthand. -->
<!-- e.g. "- inngest/inngest#123" -->
<!-- Feel free to remove this section if there are no applicable related
links.-->
- INN-
---------
Co-authored-by: jakobevangelista <jakobevangelista@gmail.com>
AndyInternet
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adds comprehensive documentation for the new
group.experiment()API, which enables running A/B and feature experiments within durable functions. It also updates the navigation structure to surface this documentation in the TypeScript reference. The documentation covers usage, configuration options, selection strategies, advanced examples, and observability features.Key changes:
Documentation for Experiments API:
group-experiment.mdxdescribing thegroup.experiment()API, including detailed usage examples, configuration options, built-in selection strategies (fixed,weighted,bucket,custom), advanced patterns (multi-step variants, multiple experiments per function), and observability details.Navigation Update:
group.experiment()documentation, marked as "new".