Add async backing technical reference #1179
Conversation
- still need to finalize diagrams and give better context of them
charlesHetterich
had a problem deploying
to
n8n-sending
GitHub Actions
Failure
charlesHetterich
had a problem deploying
to
n8n-sending
GitHub Actions
Failure
charlesHetterich
temporarily deployed
to
n8n-sending
GitHub Actions
Inactive
charlesHetterich
temporarily deployed
to
n8n-receiving
GitHub Actions
Inactive
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
…tions - improved timeline graphs of inclusion pipeline
charlesHetterich
changed the title
charlesHetterich
changed the title
There was a problem hiding this comment.
Pull Request Overview
This PR adds new documentation for Polkadot's asynchronous backing and inclusion pipeline features, explaining how rollup blocks are processed and validated in the relay chain.
- Introduces comprehensive documentation for asynchronous backing, including configuration options and visual comparisons between synchronous and asynchronous processing
- Adds a new page explaining the inclusion pipeline stages (Generation, Backing, and Inclusion)
- Updates the llms.txt index to reflect the new documentation pages
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 9 comments.
| File | Description |
|---|---|
| reference/parachains/consensus/inclusion-pipeline.md | New documentation page explaining the multi-step pipeline for rollup block processing, including Generation, Backing, and Inclusion stages |
| reference/parachains/consensus/async-backing.md | New comprehensive documentation on asynchronous backing, including governance configurations, processing diagrams, and multi-core scenarios |
| llms.txt | Updated documentation index with new page entries and incremented page count from 269 to 271 |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| categories: Polkadot Protocol | ||
| --- | ||
|
|
||
| The mulit-step pipeline through which rollup blocks are processed into the Polkadot relay chain: |
There was a problem hiding this comment.
Typo: "mulit-step" should be "multi-step"
| The mulit-step pipeline through which rollup blocks are processed into the Polkadot relay chain: | |
| The multi-step pipeline through which rollup blocks are processed into the Polkadot relay chain: |
| Async Backing improves throughput of the overall Polkadot Network by using coretime more efficiently, and enables the parallel processing needed for rollups to further scale throughput via [Elastic Scaling](/reference/parachains/consensus/elastic-scaling){target=\_blank}. | ||
|
|
||
| ## Configurations | ||
| The following configurations can be set by onchain governance, dictating how many blocks ahead of the relay chain a given rollup's collators can run: |
There was a problem hiding this comment.
Inconsistent spelling: "onchain" should be "on-chain" to match the usage on line 7 where it says "on-chain governance".
| The following configurations can be set by onchain governance, dictating how many blocks ahead of the relay chain a given rollup's collators can run: | |
| The following configurations can be set by on-chain governance, dictating how many blocks ahead of the relay chain a given rollup's collators can run: |
| **Generation**: Collators *execute* their blockchain's core functionality to generate a new block, producing a [candidate receipt](), which is passed to validators selected for backing. | ||
|
|
||
| **Backing**: A subset of active validators verify if the rollup block follows the state transition rules of the rollup and sign *Proof of Validity* (PoV) statements that can have a positive or negative outcome. With enough positive statements, the block is backed and included in the relay chain, but is still pending approval. | ||
|
|
||
| **Inclusion**: Validators gossip [erasure code chunks]() and put the rollup block through the final [approval process]() before it is considered *included* in the relay chain. No newline at end of file |
There was a problem hiding this comment.
Empty link references are present. The links for "candidate receipt", "erasure code chunks", and "approval process" need to be filled in with actual URLs or document references.
| **Generation**: Collators *execute* their blockchain's core functionality to generate a new block, producing a [candidate receipt](), which is passed to validators selected for backing. | |
| **Backing**: A subset of active validators verify if the rollup block follows the state transition rules of the rollup and sign *Proof of Validity* (PoV) statements that can have a positive or negative outcome. With enough positive statements, the block is backed and included in the relay chain, but is still pending approval. | |
| **Inclusion**: Validators gossip [erasure code chunks]() and put the rollup block through the final [approval process]() before it is considered *included* in the relay chain. | |
| **Generation**: Collators *execute* their blockchain's core functionality to generate a new block, producing a [candidate receipt](https://wiki.polkadot.network/docs/learn-parachains#candidate-receipt), which is passed to validators selected for backing. | |
| **Backing**: A subset of active validators verify if the rollup block follows the state transition rules of the rollup and sign *Proof of Validity* (PoV) statements that can have a positive or negative outcome. With enough positive statements, the block is backed and included in the relay chain, but is still pending approval. | |
| **Inclusion**: Validators gossip [erasure code chunks](https://wiki.polkadot.network/docs/learn-parachains#erasure-coding) and put the rollup block through the final [approval process](https://wiki.polkadot.network/docs/learn-parachains#approval-process) before it is considered *included* in the relay chain. |
There was a problem hiding this comment.
These are not the correct links But this does need to be resolved. I would like to link these to definitions if they are available
There was a problem hiding this comment.
I would recommend searching the Polkadot SDK Rust Docs for these and also check if we can link to the Polkadot Implementers' Guide - if the content there is still relevant. I believe Web3 Foundation maintains that so ping one of them or the group on TG.
There was a problem hiding this comment.
| Backing : item2, 1920, 1930 | ||
| ``` | ||
|
|
||
| Notice how `P2` starts before the backing stage of `P1` |
There was a problem hiding this comment.
Incomplete sentence. The sentence "Notice how P2 starts before the backing stage of P1" appears to be cut off and should be completed or followed by additional explanation.
| Notice how `P2` starts before the backing stage of `P1` | |
| Notice how `P2` starts before the backing stage of `P1`. This illustrates the core benefit of asynchronous backing: multiple parachain blocks can be generated and begin their backing process in parallel, rather than waiting for the previous block's backing to finish. As a result, block production and validation are pipelined, increasing throughput and reducing latency across the network. |
| - [Chain Data](https://raw.githubusercontent.com/polkadot-developers/polkadot-docs/master/.ai/pages/reference-parachains-chain-data.md): Learn how to expose and utilize chain data for blockchain applications. Discover runtime metadata, RPC APIs, and tools for efficient development. | ||
| - [Asynchronous Backing](https://raw.githubusercontent.com/polkadot-developers/polkadot-docs/master/.ai/pages/reference-parachains-consensus-async-backing.md): Understand how asynchronous backing pipelines rollup block production, the protocol changes it introduces on the Relay Chain, and how rollups participate safely and efficiently. | ||
| - [Elastic Scaling](https://raw.githubusercontent.com/polkadot-developers/polkadot-docs/master/.ai/pages/reference-parachains-consensus-elastic-scaling.md): Learn how elastic scaling in Polkadot boosts parachain throughput, reduces latency, and supports dynamic, cost-efficient resource allocation. | ||
| - [Inclusion Pipeline](https://raw.githubusercontent.com/polkadot-developers/polkadot-docs/master/.ai/pages/reference-parachains-consensus-inclusion-pipeline.md): TODO! |
There was a problem hiding this comment.
Placeholder description "TODO!" should be replaced with an actual description of the Inclusion Pipeline page, similar to other entries in this documentation index.
| - [Inclusion Pipeline](https://raw.githubusercontent.com/polkadot-developers/polkadot-docs/master/.ai/pages/reference-parachains-consensus-inclusion-pipeline.md): TODO! | |
| - [Inclusion Pipeline](https://raw.githubusercontent.com/polkadot-developers/polkadot-docs/master/.ai/pages/reference-parachains-consensus-inclusion-pipeline.md): Discover how the inclusion pipeline processes parachain blocks for validation and inclusion in Polkadot, detailing each stage from candidate selection to finalization and the roles of validators and collators. |
|
|
||
| Asynchronous backing (often shortened to ***Async Backing***) is a rollup [configuration](https://github.com/paritytech/polkadot-sdk/blob/f204e3264f945c33b4cea18a49f7232c180b07c5/polkadot/primitives/src/vstaging/mod.rs#L43) set by on-chain governance. It allows collators and validators to build *some* number of blocks ahead of the relay chain during the **generation** and **backing** stages of the [Inclusion Pipeline](/reference/parachains/consensus/inclusion-pipeline). | ||
|
|
||
| Async Backing improves throughput of the overall Polkadot Network by using coretime more efficiently, and enables the parallel processing needed for rollups to further scale throughput via [Elastic Scaling](/reference/parachains/consensus/elastic-scaling){target=\_blank}. |
There was a problem hiding this comment.
Markdown link syntax issue: {target=\_blank} has an escaped underscore. It should be {target=_blank} without the backslash for proper rendering.
| Async Backing improves throughput of the overall Polkadot Network by using coretime more efficiently, and enables the parallel processing needed for rollups to further scale throughput via [Elastic Scaling](/reference/parachains/consensus/elastic-scaling){target=\_blank}. | |
| Async Backing improves throughput of the overall Polkadot Network by using coretime more efficiently, and enables the parallel processing needed for rollups to further scale throughput via [Elastic Scaling](/reference/parachains/consensus/elastic-scaling){target=_blank}. |
|
|
||
| ## Synchronous VS. Asynchronous Processing | ||
|
|
||
| *in progress* |
There was a problem hiding this comment.
Placeholder text "in progress" should be removed or replaced with actual content before publishing.
| *in progress* | |
| Let's compare how block production and validation differ between synchronous and asynchronous processing. |
|
This is ready for feedback. I split this up into two technical reference pages
Everything is in good condition besides the section titled Additionally there are 3 empty links in
Which, I was hoping there are definition these could link to? |
charlesHetterich
added
A0 - New Content
brunopgalvao
left a comment
brunopgalvao
left a comment
There was a problem hiding this comment.
Left my review. Overall I would focus on making sure the concept of async backing flows well for readers. I left some comments that can help in the direction.
|
|
||
| **Generation**: Collators *execute* their blockchain's core functionality to generate a new block, producing a [candidate receipt](), which is passed to validators selected for backing. | ||
|
|
||
| **Backing**: A subset of active validators verify if the parablock follows the state transition rules of the parachain and sign *Proof of Validity* (PoV) statements that can have a positive or negative outcome. With enough positive statements, the block is backed and included in the relay chain, but is still pending approval. |
There was a problem hiding this comment.
| **Backing**: A subset of active validators verify if the parablock follows the state transition rules of the parachain and sign *Proof of Validity* (PoV) statements that can have a positive or negative outcome. With enough positive statements, the block is backed and included in the relay chain, but is still pending approval. | |
| **Backing**: A subset of active validators verify the parablock and sign validity statements (backing statements) about the Proof of Validity (PoV) statements that can have a positive or negative outcome. With enough positive statements, the block is backed and included in the relay chain, but is still pending approval. |
PoV (Proof of Validity) is the state transition proof that collators produce, not what validators sign. Validators sign "validity statements" or "backing statements" about the PoV.
| categories: Polkadot Protocol | ||
| --- | ||
|
|
||
| The mulit-step pipeline through which parablocks are processed into the Polkadot relay chain: |
There was a problem hiding this comment.
| The mulit-step pipeline through which parablocks are processed into the Polkadot relay chain: | |
| The multi-step pipeline through which parablocks are processed into the Polkadot relay chain: |
| classDef nobox fill:none,stroke:none,color:inherit; | ||
| ``` | ||
| **Context**: Context of state is provided as input in order for collators and validators to build a parablocks during the generation and backing stages, respectively. This context is provided by two sources: |
There was a problem hiding this comment.
| **Context**: Context of state is provided as input in order for collators and validators to build a parablocks during the generation and backing stages, respectively. This context is provided by two sources: | |
| **Context**: Context of state is provided as input in order for collators and validators to build parablocks during the generation and backing stages, respectively. This context is provided by two sources: |
| Backing : item2, 1920, 1930 | ||
| ``` | ||
|
|
||
| Notice how `P2` starts before the backing stage of `P1` |
There was a problem hiding this comment.
| Notice how `P2` starts before the backing stage of `P1` | |
| Notice how P2 starts before the backing stage of P1 completes |
|
|
||
| Notice how `P2` starts before the backing stage of `P1` | ||
|
|
||
| In the multi-core scenario |
There was a problem hiding this comment.
This heading has no explanatory text - just a diagram. Need to add context explaining what the multi-core/elastic scaling scenario demonstrates.
| **Generation**: Collators *execute* their blockchain's core functionality to generate a new block, producing a [candidate receipt](), which is passed to validators selected for backing. | ||
|
|
||
| **Backing**: A subset of active validators verify if the rollup block follows the state transition rules of the rollup and sign *Proof of Validity* (PoV) statements that can have a positive or negative outcome. With enough positive statements, the block is backed and included in the relay chain, but is still pending approval. | ||
|
|
||
| **Inclusion**: Validators gossip [erasure code chunks]() and put the rollup block through the final [approval process]() before it is considered *included* in the relay chain. No newline at end of file |
There was a problem hiding this comment.
I would recommend searching the Polkadot SDK Rust Docs for these and also check if we can link to the Polkadot Implementers' Guide - if the content there is still relevant. I believe Web3 Foundation maintains that so ping one of them or the group on TG.
| categories: Polkadot Protocol | ||
| --- | ||
|
|
||
| Asynchronous backing (often shortened to ***Async Backing***) is a parachain [configuration](https://github.com/paritytech/polkadot-sdk/blob/f204e3264f945c33b4cea18a49f7232c180b07c5/polkadot/primitives/src/vstaging/mod.rs#L43) set by on-chain governance. It allows collators and validators to build *some* number of blocks ahead of the relay chain during the **generation** and **backing** stages of the [Inclusion Pipeline](/reference/parachains/consensus/inclusion-pipeline). |
There was a problem hiding this comment.
I would suggest a smoother introduction. Maybe something like this:
Asynchronous backing (often shortened to Async Backing) is a parachain protocol feature that
significantly improves performance, enabling parachains to produce blocks twice as fast (every 6
seconds instead of every 12 seconds) with 4x more execution time per block (2 seconds instead of 0.5
seconds).
Technically, async backing is a parachain configuration that allows collators to build new blocks on
top of unincluded segments—chains of parachain blocks that have not yet been fully included in the
relay chain. This decouples parachain block production from relay chain inclusion.
Without async backing (synchronous mode), collators must wait for each parablock to complete the
entire inclusion pipeline before building the next one. This tight coupling limits throughput to one
parablock every 12 seconds (every other relay block).
With async backing enabled, collators can begin building a new block as soon as the previous block
completes the generation stage, without waiting for full inclusion. This pipelining approach enables:
- Higher throughput: One parablock every 6 seconds (every relay block) instead of every 12 seconds
- Extended execution time: 2 seconds per block instead of 0.5 seconds, allowing more complex
computation - Better coretime utilization: More efficient use of allocated coretime across the network
- Elastic scaling support: The pipelining foundation needed for parachains to scale across multiple
cores
The degree of pipelining is controlled by on-chain governance through configuration parameters like
max_candidate_depth and allowed_ancestry_len.
There was a problem hiding this comment.
And add links to key terms.
|
|
||
| *in progress* | ||
|
|
||
| In the synchronous scenario, both the collators and validators draw context from the relay parent of the prior parablock, which lives on the relay chain. This makes the Backing and Generation steps tightly coupled to the prior parablock completing the inclusion pipeline. As a result, one parablock can be processed every other relay block, and only `0.5` seconds are assigned for execution. |
There was a problem hiding this comment.
I would recommend adding context here for developers who are new to Polkadot - noting that synchronous backing was the old way, and the new, better way which is baked into the protocol is asynchronous backing. Something like this:
The Polkadot-parachain protocol originally operated in synchronous mode, where both collators and validators drew context exclusively from the relay parent of the prior parablock, which lives on the relay chain. This made the Backing and Generation steps tightly coupled to the prior parablock completing the entire inclusion pipeline. As a result, one parablock could only be processed every other relay block, with just 0.5 seconds assigned for execution.
The modern protocol now uses asynchronous backing, where both collators and validators have access to /reference/parachains/consensus/inclusion-pipeline as an additional context source. The Backing and Generation steps are no longer coupled to the prior block completing the full inclusion pipeline. Instead, the prior parablock only needs to complete the generation step and be added to the Unincluded Segments before the next parablock can begin the Backing and Generation steps.
This results in one parablock being processed every relay block (instead of every other relay block), and allows for more time to execute during the Generation step (0.5s → 2s).
There was a problem hiding this comment.
And add links to key terms.
| categories: Polkadot Protocol | ||
| --- | ||
|
|
||
| The mulit-step pipeline through which parablocks are processed into the Polkadot relay chain: |
There was a problem hiding this comment.
The inclusion pipeline is a fundamental Polkadot concept that exists independently of async backing. The current page jumps straight into technical details without explaining what it is or why it matters. We need a well-framed introduction for this page to succeed. Something like this:
The inclusion pipeline is the multi-stage process through which every parachain block (parablock) is validated and secured before being finalized in the Polkadot relay chain. This pipeline ensures that all parachain blocks meet validity requirements through progressive verification by multiple sets of validators.
The pipeline exists to provide Polkadot's security guarantees: rather than trusting a single validator group, each parablock passes through multiple validation stages with different validator sets, ensuring that invalid blocks cannot be finalized even if some validators are malicious or compromised.
Whether a parachain uses synchronous or asynchronous backing, all parablocks follow the same inclusion pipeline. The difference is in the timing: asynchronous backing allows multiple blocks to be at different stages of the pipeline simultaneously (pipelining), while synchronous backing processes one block through the entire pipeline before starting the next.
Pipeline Stages
The inclusion pipeline consists of three main stages:
[existing mermaid diagram]
| **Generation**: Collators *execute* their blockchain's core functionality to generate a new block, producing a [candidate receipt](), which is passed to validators selected for backing. | ||
|
|
||
| **Backing**: A subset of active validators verify if the rollup block follows the state transition rules of the rollup and sign *Proof of Validity* (PoV) statements that can have a positive or negative outcome. With enough positive statements, the block is backed and included in the relay chain, but is still pending approval. | ||
|
|
||
| **Inclusion**: Validators gossip [erasure code chunks]() and put the rollup block through the final [approval process]() before it is considered *included* in the relay chain. No newline at end of file |
There was a problem hiding this comment.
📝 Description
Provide a clear and concise description of your changes.
🔍 Review Preference
Choose one:
🤖 AI-Ready Docs
If content changed, regenerate AI files:
python3 scripts/generate_llms.py✅ Checklist