CI: Test new runtime compatibility with latest client #1517

ipapandinas · 2025-07-22T09:16:30Z

Pull Request Summary

Closes #1310

Tested here https://github.com/AstarNetwork/Astar/actions/runs/16443086835 with a fake runtime upgrade 1602

ashutoshvarma

Nice!
LGTM

ermalkaleci

This needs to be separated from release workflow. There's no need for this to run before release note.

Dinonard

Looks good!

Please just try not to introduce any additional possible failure points for the draft release process.

.github/workflows/release-runtime.yml

Dinonard · 2025-07-23T07:07:52Z

.github/workflows/release-runtime.yml

            ${{ matrix.chain }}-metadata.json
            ${{ matrix.chain }}-diff.txt

+  fetch-github-releases:


Is there a need to do this as a separate step?

If previous step for fetching last runtime release is kept where it was, we have zero impact on the existing CI. With this change, you're introducing a new possibility of the whole release failing because e.g. if fails to fetch the last client release. Realistically, release can be made without that information.

I'd suggest to keep this as it was before, and move client fetching to the new task you've added for regression testing.

All three chains matrix jobs need the same client binary, it's much more efficient to fetch it once in a separate job. Now that I've added a separate workflow, there are no changes to the original Runtime Build one.

Dinonard · 2025-07-23T07:11:01Z

third-party/zombienet/smoke_runtime_upgrade.zndsl

+collator2: reports block height is greater than 5 within 200 seconds
+
+## perform runtime upgrade
+collator1: parachain 1000 perform upgrade with ./runtime.compact.compressed.wasm within 200 seconds


Won't this step fail if runtime upgrade fails?

Asking since I'm wondering if the script is needed.

I've tested to upgrade using the same spec version and this step did not failed, I prefer to keep the JS script.

Upgrade failed but step succeeded?

Yes, when done manually with a local Zombienet network, the upgrade fails with a SpecVersionNeedsToIncrease error. But when performed via smoke testing, the step succeeded (check the attached screenshot below, the JS script caught it).

However, in the case of an uplift with a client + runtime release, we could be in a situation where the new runtime is checked against the new client, and I don’t want false positives to be emitted, so I’ll remove the JS script.

I think Ermal raised a good point here, it’s better to check compatibility against the two latest clients. I reproduced the issue we had with host functions using version v5.41.0 and the upgrade step failed during Zombienet testing with runtime 91 (see the second screenshot below). But not with v5.42.0 for the first reason explained (client have runtime 91 in it).

Ok, but the upgrade didn't fail in that case - it just replaced runtime with the one that has the same spec version (and is probably identical).

Maybe for more context, why I asked - it seems strange to me that a DSL exists for testing, a dedicated command for runtime upgrade exists, but it can silently fail. This is why your script seemed redundant to me.

For checking last two clients, it seems a bit too much to me, but if you think it's the best, do it.

Ok, thanks for the extra context, it has been removed now.

ipapandinas · 2025-07-24T08:01:51Z

@Dinonard @ermalkaleci - Thank you for your feedbacks, I've separated this check into a Runtime Compatibility workflow that is triggered after the original Runtime Build one is completed (or manually if a Runtime Build worflow ID is provided to download artifacts from)

third-party/zombienet/setup.sh

ermalkaleci · 2025-07-24T08:22:16Z

.github/workflows/runtime-compatibility.yml

+            sleep $((i * 5))
+          done
+          
+          # Get latest client tag (v*.*.* format)


not sure about this, the idea is to test new runtime with old clients, if we release new client then the test will always pass

Is there something better we can do though?
Test last two releases? Seems like an overkill.

Good point @ermalkaleci, the workflow checks compatibility for the 2 latest clients now. Since they're non-gating checks, this only provides extra infos but could be beneficial as explained here.

Dinonard · 2025-07-24T14:41:37Z

.github/workflows/runtime-compatibility.yml

+          # Get latest client info
+          CLIENT_TAG_1="${CLIENT_TAGS_ARRAY[0]}"
+          ASSET_NAME_1="astar-collator-${CLIENT_TAG_1}-ubuntu-x86_64.tar.gz"
+          CLIENT_ASSET_URL_1=$(curl -s "https://api.github.com/repos/AstarNetwork/Astar/releases/tags/${CLIENT_TAG_1}" | \
+          jq -r --arg name "$ASSET_NAME_1" '.assets[] | select(.name == $name) | .browser_download_url')
+ 
+          # Get second latest client info
+          CLIENT_TAG_2="${CLIENT_TAGS_ARRAY[1]}"
+          ASSET_NAME_2="astar-collator-${CLIENT_TAG_2}-ubuntu-x86_64.tar.gz"
+          CLIENT_ASSET_URL_2=$(curl -s "https://api.github.com/repos/AstarNetwork/Astar/releases/tags/${CLIENT_TAG_2}" | \
+          jq -r --arg name "$ASSET_NAME_2" '.assets[] | select(.name == $name) | .browser_download_url')


I'm no bash expert, but this can be a loop, and you can transfer values via evn variables like an array 🙂.

And to expand on that, the job that tests the last two clients - you can change it to spin up collator using two different binaries. Anyways, just a suggestion if that's something you'd like to try.

This is a good suggestion, it reduces the matrix size and reflects a more realistic scenario where collators use different client versions within the same network. I had to slightly adjust the network setup files.
Also with the for loops code is somehow more DRY.

I've tested here: https://github.com/ipapandinas/Astar/actions/runs/16517224780

Great!

Just one more comment from me, in zndsl file.
Otherwise LGTM.

Dinonard · 2025-07-25T08:39:11Z

third-party/zombienet/smoke_runtime_upgrade.zndsl


 ## perform runtime upgrade
-collator1: parachain 1000 perform upgrade with ./runtime.compact.compressed.wasm within 200 seconds
+collator2: parachain 1000 perform upgrade with ./runtime.compact.compressed.wasm within 200 seconds


This also needs to be done for collator1, right?

In the previous iteration with the JS script, I performed the upgrade on collator 1 and tested propagation with the script on collator 2. Now I'm upgrading collator 2 because it's the second-latest client, so in client + runtime releases, it's the one most likely to fail. I think the upgrade is automatically propagated to 1, and after that, I'm making sure it keeps building blocks. However now, I am not 100% sure with different clients, WDYT?

For some reason in my head it was for two separate networks. Ignore my incorrect comment, you are right, upgrade is propagated.

It doesn't matter which one is upgraded since it's the relaychain that sends the signal for the upgrade to go ahead.

github-actions · 2025-07-25T08:49:59Z

Package	Line Rate	Branch Rate	Health
precompiles/dapp-staking/src/test	0%	0%	❌
pallets/dapp-staking/src	80%	0%	✔
pallets/collator-selection/src	89%	0%	✔
pallets/dapp-staking/src/benchmarking	95%	0%	✔
precompiles/assets-erc20/src	77%	0%	➖
precompiles/dispatch-lockdrop/src	83%	0%	✔
pallets/collective-proxy/src	94%	0%	✔
pallets/democracy-mbm/src	30%	0%	❌
pallets/xc-asset-config/src	62%	0%	➖
pallets/dapp-staking/rpc/runtime-api/src	0%	0%	❌
precompiles/xcm/src	69%	0%	➖
chain-extensions/unified-accounts/src	0%	0%	❌
pallets/vesting-mbm/src	87%	0%	✔
pallets/unified-accounts/src	80%	0%	✔
pallets/ethereum-checked/src	76%	0%	➖
chain-extensions/types/assets/src	0%	0%	❌
pallets/inflation/src	84%	0%	✔
pallets/astar-xcm-benchmarks/src/fungible	100%	0%	✔
pallets/astar-xcm-benchmarks/src/generic	100%	0%	✔
pallets/dynamic-evm-base-fee/src	84%	0%	✔
precompiles/sr25519/src	56%	0%	➖
chain-extensions/pallet-assets/src	54%	0%	➖
primitives/src/xcm	64%	0%	➖
chain-extensions/types/unified-accounts/src	0%	0%	❌
primitives/src	54%	0%	➖
pallets/price-aggregator/src	75%	0%	➖
pallets/static-price-provider/src	91%	0%	✔
precompiles/substrate-ecdsa/src	67%	0%	➖
precompiles/dapp-staking/src	89%	0%	✔
precompiles/unified-accounts/src	100%	0%	✔
pallets/astar-xcm-benchmarks/src	86%	0%	✔
pallets/dapp-staking/src/test	0%	0%	❌
Summary	74% (3791 / 5116)	0% (0 / 0)	➖

Minimum allowed line rate is 50%

Dinonard

LGTM

Seems new release for zepter is causing problems though.

ipapandinas added 9 commits July 21, 2025 11:48

wip

fa65c79

tmp job disabling

7bc6529

wip2

0878a9c

fake version

2deb0bb

check code params extraction

c52deca

zombienet test

714c36b

fix

7823c97

add checkout

99370c0

test full workflow

1038f62

ipapandinas added the ci This PR/Issue is related to the topic "CI" label Jul 22, 2025

ipapandinas added 4 commits July 22, 2025 12:15

make a separate job to fetch GH releases

9f73ad2

add retry

8db94d8

add retry to zombienet

1c4074f

cleanup

1a15cc3

ipapandinas marked this pull request as ready for review July 22, 2025 12:34

ipapandinas requested review from Dinonard, PierreOssun, ashutoshvarma and ermalkaleci July 22, 2025 12:37

ashutoshvarma previously approved these changes Jul 23, 2025

View reviewed changes

ermalkaleci reviewed Jul 23, 2025

View reviewed changes

Dinonard reviewed Jul 23, 2025

View reviewed changes

ipapandinas added 2 commits July 23, 2025 23:17

separate workflows

aaf4200

fake runtime version for tests

ef220f7

ipapandinas dismissed ashutoshvarma’s stale review via ef220f7 July 23, 2025 21:20

ipapandinas added 5 commits July 23, 2025 23:34

tests

fd89277

more tests

53519dc

more testss

fa3dea0

small fix

5417c3f

more fixes

ebce060

ipapandinas added 8 commits July 24, 2025 00:02

test zombienet step fails if runtime upgrade fails

977c053

test trigger

dadb181

support manual trigger

cd3969f

cleanup

74bc032

more cleanup

a219db7

more cleanups

5cc177d

newline

fc438e4

debug step removed

7f14bad

ermalkaleci reviewed Jul 24, 2025

View reviewed changes

third-party/zombienet/setup.sh Outdated Show resolved Hide resolved

ermalkaleci reviewed Jul 24, 2025

View reviewed changes

latest 2 clients

e15b624

Dinonard reviewed Jul 24, 2025

View reviewed changes

single network with mixed collator clients

8f34f27

Dinonard reviewed Jul 25, 2025

View reviewed changes

Dinonard approved these changes Jul 25, 2025

View reviewed changes

ermalkaleci approved these changes Jul 28, 2025

View reviewed changes

ipapandinas merged commit ce3e1a0 into master Jul 28, 2025
7 of 8 checks passed

ipapandinas deleted the ci/test-legacy-client branch July 28, 2025 12:03

CI: Test new runtime compatibility with latest client #1517

CI: Test new runtime compatibility with latest client #1517

Uh oh!

Conversation

ipapandinas commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashutoshvarma left a comment

Choose a reason for hiding this comment

Uh oh!

ermalkaleci left a comment

Choose a reason for hiding this comment

Uh oh!

Dinonard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ipapandinas Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ipapandinas commented Jul 24, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ipapandinas Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 25, 2025

Uh oh!

Dinonard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ipapandinas commented Jul 22, 2025 •

edited

Loading

ipapandinas Jul 24, 2025 •

edited

Loading

ipapandinas Jul 24, 2025 •

edited

Loading