Problems running multi_parachains.toml configuration using zombienet

[d0na]

Description

I am encountering issues when trying to run zombienet with the multi_parachains.toml configuration file located in ./third-party/zombienet.

The problem occurs in both this two scenarios:

1. Using Polkadot Docker Image in the multi_parachains.toml

default_image = "paritypr/polkadot-debug:master"
default_command = "./polkadot"

2. Using a local version of Polkadot in the multi_parachains.toml . This version was compiled using a MacBookPro arm64 version.

default_command = "./polkadot" # version 1.17.0-569ce71e2c7

with the following error

Error:  	 Error: Command failed with exit code 1: bash -c ./bin/polkadot build-spec --chain /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-361ecf9d58647f20f5871b7d60904394_-17503-J7Jh24BwqF2x/rococo-local-plain.json --disable-default-bootnode  --raw > /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-361ecf9d58647f20f5871b7d60904394_-17503-J7Jh24BwqF2x/rococo-local-raw.json
2025-01-23 14:16:07 Building chain spec
2025-01-23 14:16:08 Took active validators from set with wrong size
2025-01-23 14:16:08 Took active validators from set with wrong size
2025-01-23 14:16:08 Took active validators from set with wrong size.
2025-01-23 14:16:08 Took active validators from set with wrong size
2025-01-23 14:16:08 panicked at /Users/francesco/workspace/git/PHD/lab/polkadot-sdk/polkadot/runtime/parachains/src/hrmp.rs:909:4:
failed to initialize the genesis storage: Module(ModuleError { index: 60, error: [1, 0, 0, 0], message: Some("OpenHrmpChannelInvalidRecipient") })
Error:
   0: Other: wasm call error Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
      WASM backtrace:
      error while executing at wasm backtrace:
          0: 0x708523 - rococo_runtime.wasm!rust_begin_unwind
          1: 0xee0e - rococo_runtime.wasm!core::panicking::panic_fmt::h697e3d119f551db3
          2: 0x6aa03f - rococo_runtime.wasm!polkadot_runtime_parachains::hrmp::initialize_storage::he9bea7bee3f5beff
          3: 0xa2ad3 - rococo_runtime.wasm!<rococo_runtime::RuntimeGenesisConfig as frame_support::traits::hooks::BuildGenesisConfig>::build::h2e20a486c5d31e33
          4: 0x5a8b38 - rococo_runtime.wasm!frame_support::genesis_builder_helper::build_state::h2355dcc9bf4f2368
          5: 0x391273 - rococo_runtime.wasm!GenesisBuilder_build_state

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Steps to Reproduce

cd third-party/zombienet

#then configure the multi_parachains.toml using the polkadot versions used above

#For the docker version use 
#default_image = "paritypr/polkadot-debug:master"
#default_command = "./polkadot"

#For the local version use 
#default_command = "./polkadot"


zombienet -p native spawn multi_parachains.toml

Expected

• The configuration filemulti_parachains.tomlshould execute successfully without errors.
• The zombienet command should generate througth polkadot the relay chain specifications (rococo-local) without any issues.
• zombienet should initialize the parachains with evm envinroment and xcm support with preconfigured channels

Environment

• Operating system: Sonoma 14.3.1
• Computer: MAcBookPro nov.- 2023 - M3 Pro
• Project version/tag: latest
• Rust version (run rustup show):
• Polkadot local version : 1.17.0-569ce71e2c7
• Polkadot image version: paritypr/polkadot-debug:master
• zombienet version: 1.3.118

Default host: aarch64-apple-darwin
rustup home:  /Users/francesco/.rustup

installed toolchains
--------------------

stable-aarch64-apple-darwin (default)
nightly-aarch64-apple-darwin
1.77.0-aarch64-apple-darwin

installed targets for active toolchain
--------------------------------------

aarch64-apple-darwin
wasm32-unknown-unknown

active toolchain
----------------

stable-aarch64-apple-darwin (default)
rustc 1.84.0 (9fc6b4312 2025-01-07)

Logs, Errors

Provider : native
Namespace : zombie-3d552a19d7ace91f936c2a777c19f0fc
Temp Dir : /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32
Pod : temp
Status : Launching
Command : bash -c ./bin/polkadot build-spec --chain rococo-local --disable-default-bootnode > / var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-2 0728-nLh8TuFVjt32/cfg/rococo-local-plain.json
Pod : temp
Status : Ready
Pod : temp-1
Status : Launching
Command : bash -c ../../../Astar/target/production/astar-collator build-spec --chain shibuya-de v --disable-default-bootnode > /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d5 52a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/cfg/shibuya-dev-plain.json
Pod : temp-2
Status : Launching
Command : bash -c ../../../Astar/target/production/astar-collator build-spec --chain shiden-dev  --disable-default-bootnode > /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d55 2a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/cfg/shiden-dev-plain.json
Pod : temp-2
Status : Ready
Pod : temp-1
Status : Ready
🧹 Starting with a fresh authority set...
👤 Added Genesis Authority : collator2 : 5HdWkjfNLeJ4LQXxP3t6ZzcWHVwWudLvF9SUPAysbqZBW4Qw
👤 Added CollatorSelection  : collator2 : 5E5E37FNZD9KVHyGgSHt8pc2kq8e3VUS5rf8GmrxCa7ySs8s
Pod : temp-3
Status : Launching
Command : bash -c ../../../Astar/target/production/astar-collator build-spec --chain /var/folde rs/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8 TuFVjt32/shiden-dev-2007-rococo-local-plain.json --disable-default-bootnode  --raw > /var/ folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728 -nLh8TuFVjt32/shiden-dev-2007-rococo-local-raw.json
🧹 Starting with a fresh authority set...
👤 Added Genesis Authority : collator1 : 5FCnz6Sb9ue73TQzrwVLkwT5ZAr3TUQEzJSMdWCP2D4zf1t3
👤 Added CollatorSelection  : collator1 : 5Do5UoayFvDrHroGS1YMqxTVUysSkrhNwVMzmj1foVb3vzzb
Pod : temp-4
Status : Launching
Command : bash -c ../../../Astar/target/production/astar-collator build-spec --chain /var/folde rs/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8 TuFVjt32/shibuya-dev-1000-rococo-local-plain.json --disable-default-bootnode  --raw > /var /folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-2072 8-nLh8TuFVjt32/shibuya-dev-1000-rococo-local-raw.json
Pod : temp-3
Status : Ready
Pod : temp-4
Status : Ready
Pod : temp-collator
Status : Launching
Command : bash -c ../../../Astar/target/production/astar-collator export-genesis-state -d /var/ folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728 -nLh8TuFVjt32/export-genesis-state/2007 --chain /var/folders/66/gvjn_skn1dl9f5gns8s3prth00 00gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/shiden-dev-2007-rococo- local.json /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a7 77c19f0fc_-20728-nLh8TuFVjt32/cfg/genesis-state-2007 && ../../../Astar/target/production/a star-collator export-genesis-wasm --chain /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T /zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/shiden-dev-2007-rococo-local. json /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c19f 0fc_-20728-nLh8TuFVjt32/cfg/genesis-wasm-2007
Pod : temp-collator-1
Status : Launching
Command : bash -c ../../../Astar/target/production/astar-collator export-genesis-state -d /var/ folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728 -nLh8TuFVjt32/export-genesis-state/1000 --chain /var/folders/66/gvjn_skn1dl9f5gns8s3prth00 00gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/shibuya-dev-1000-rococo -local.json /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a 777c19f0fc_-20728-nLh8TuFVjt32/cfg/genesis-state-1000 && ../../../Astar/target/production/ astar-collator export-genesis-wasm --chain /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/ T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/shibuya-dev-1000-rococo-loca l.json /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c1 9f0fc_-20728-nLh8TuFVjt32/cfg/genesis-wasm-1000
Pod : temp-collator
Status : Ready
Pod : temp-collator-1
Status : Ready
✓ Added Genesis Parachain 1000
✓ Added Genesis Parachain 2007
🧹 Starting with a fresh authority set...
👤 Added Balance 2000000000000 for alice - 5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY
👤 Added Balance 2000000000000 for bob - 5HpG9w8EBLe5XCrbczpwq5TSXvedjrBGCwqxK1iQ7qUsSWFc
👤 Added Balance 2000000000000 for charlie - 5Ck5SLSHYac6WFt5UZRSsdJjwmpSZq85fd5TRNAdZQVzEAPT
👤 Added Balance 2000000000000 for dave - 5HKPmK9GYtE1PSLsS1qiYU9xQ9Si1NcEhdeCq9sw5bqu4ns8
👤 Added Genesis Authority : alice : 5GNJqTPyNqANBkUVMN1LPPrxXnFouWXoe2wNSmmEoLctxiZY
👤 Added Genesis Authority : bob : 5HpG9w8EBLe5XCrbczpwq5TSXvedjrBGCwqxK1iQ7qUsSWFc
👤 Added Genesis Authority : charlie : 5Ck5SLSHYac6WFt5UZRSsdJjwmpSZq85fd5TRNAdZQVzEAPT
👤 Added Genesis Authority : dave : 5HKPmK9GYtE1PSLsS1qiYU9xQ9Si1NcEhdeCq9sw5bqu4ns8

	 Adding Genesis HRMP Channels
✓ Added HRMP channel 2000 -> 2007
✓ Added HRMP channel 2007 -> 2000
Pod : temp-5
Status : Launching
Command : bash -c ./bin/polkadot build-spec --chain /var/folders/66/gvjn_skn1dl9f5gns8s3prth000 0gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/rococo-local-plain.json  --disable-default-bootnode  --raw > /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombi e-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/rococo-local-raw.json
Error:  	 Error: Command failed with exit code 1: bash -c ./bin/polkadot build-spec --chain /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/rococo-local-plain.json --disable-default-bootnode  --raw > /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/rococo-local-raw.json
2025-01-23 18:26:35 Building chain spec
2025-01-23 18:26:36 Took active validators from set with wrong size
2025-01-23 18:26:36 Took active validators from set with wrong size
2025-01-23 18:26:36 Took active validators from set with wrong size.
2025-01-23 18:26:36 Took active validators from set with wrong size
2025-01-23 18:26:36 panicked at /Users/francesco/workspace/git/PHD/lab/polkadot-sdk/polkadot/runtime/parachains/src/hrmp.rs:909:4:
failed to initialize the genesis storage: Module(ModuleError { index: 60, error: [1, 0, 0, 0], message: Some("OpenHrmpChannelInvalidRecipient") })
Error:
   0: Other: wasm call error Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
      WASM backtrace:
      error while executing at wasm backtrace:
          0: 0x708523 - rococo_runtime.wasm!rust_begin_unwind
          1: 0xee0e - rococo_runtime.wasm!core::panicking::panic_fmt::h697e3d119f551db3
          2: 0x6aa03f - rococo_runtime.wasm!polkadot_runtime_parachains::hrmp::initialize_storage::he9bea7bee3f5beff
          3: 0xa2ad3 - rococo_runtime.wasm!<rococo_runtime::RuntimeGenesisConfig as frame_support::traits::hooks::BuildGenesisConfig>::build::h2e20a486c5d31e33
          4: 0x5a8b38 - rococo_runtime.wasm!frame_support::genesis_builder_helper::build_state::h2355dcc9bf4f2368
          5: 0x391273 - rococo_runtime.wasm!GenesisBuilder_build_state

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

undefined
Node'
s logs:
 : /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/logs

 Error:  	 Error: Command failed with exit code 1: bash -c kill -9
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

[Dinonard]

I'd suggest to check the content of the output file:
Command : bash -c ./bin/polkadot build-spec --chain /var/folders/66/gvjn_skn1dl9f5gns8s3prth000 0gn/T/zombie-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/rococo-local-plain.json --disable-default-bootnode --raw > /var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombi e-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/rococo-local-raw.json

/var/folders/66/gvjn_skn1dl9f5gns8s3prth0000gn/T/zombi e-3d552a19d7ace91f936c2a777c19f0fc_-20728-nLh8TuFVjt32/rococo-local-raw.json

You can try & execute the steps to construct the chain spec manually (repeat the commands in your terminal, and observe the error you get).

[d0na]

I tried executing the command manually as you suggested.

Unfortunately, the result is the same. The file rococo-local-raw.json remains empty and I get the same error as before.

To investigate further, I added the --log debug option, and the output is as follows:

2025-01-24 08:37:09.491 DEBUG main wasmtime_jit::code_memory: ignoring section
2025-01-24 08:37:09.491 DEBUG main wasmtime_jit::code_memory: ignoring section .wasmtime.engine
2025-01-24 08:37:09.491 DEBUG main wasmtime_jit::code_memory: ignoring section .symtab
2025-01-24 08:37:09.491 DEBUG main wasmtime_jit::code_memory: ignoring section .strtab
2025-01-24 08:37:09.491 DEBUG main wasmtime_jit::code_memory: ignoring section .shstrtab
2025-01-24 08:37:09.502 DEBUG main wasm-runtime: Prepared new runtime version Some(RuntimeVersion { spec_name: "rococo", impl_name: "parity-rococo-v2.0", authoring_version: 0, spec_version: 1017001, impl_version: 0, apis: [([223, 106, 203, 104, 153, 7, 96, 155], 5), ([111, 245, 46, 232, 88, 230, 197, 189], 1), ([145, 177, 200, 177, 99, 40, 235, 146], 1), ([159, 251, 80, 90, 167, 56, 214, 156], 1), ([55, 227, 151, 252, 124, 145, 245, 228], 2), ([64, 254, 58, 212, 1, 248, 149, 154], 6), ([210, 188, 152, 151, 238, 208, 143, 21], 3), ([247, 139, 39, 139, 229, 63, 69, 76], 2), ([175, 44, 2, 151, 162, 62, 109, 61], 11), ([73, 234, 175, 27, 84, 138, 12, 176], 5), ([145, 213, 223, 24, 176, 210, 207, 88], 2), ([237, 153, 197, 172, 178, 94, 237, 245], 3), ([203, 202, 37, 227, 159, 20, 35, 135], 2), ([104, 122, 212, 74, 211, 127, 3, 194], 1), ([171, 60, 5, 114, 41, 31, 235, 139], 1), ([188, 157, 137, 144, 79, 91, 146, 63], 1), ([55, 200, 187, 19, 80, 169, 162, 168], 4), ([42, 94, 146, 70, 85, 57, 158, 96], 1), ([251, 197, 119, 185, 215, 71, 239, 214], 1), ([38, 9, 190, 131, 172, 68, 104, 220], 1)], transaction_version: 26, system_version: 1 }) in 840 ms.
2025-01-24 08:37:09.504 DEBUG main wasm-heap: allocator dropped: AllocationStats { bytes_allocated: 4120, bytes_allocated_peak: 8224, bytes_allocated_sum: 12272, address_space_used: 12272 }
2025-01-24 08:37:09.504 DEBUG main wasm-runtime: Allocated WASM instance 1/2
2025-01-24 08:37:09.548 DEBUG main runtime: No initial validator provided by `SessionManager`, use session config keys to generate initial validator set.
2025-01-24 08:37:09.548  WARN main runtime::parachains: Took active validators from set with wrong size
2025-01-24 08:37:09.548  WARN main runtime::parachains: Took active validators from set with wrong size
2025-01-24 08:37:09.548  WARN main runtime::parachains: Took active validators from set with wrong size.
2025-01-24 08:37:09.548  WARN main runtime::parachains: Took active validators from set with wrong size
2025-01-24 08:37:09.553 ERROR main runtime: panicked at /Users/francesco/workspace/git/PHD/lab/polkadot-sdk/polkadot/runtime/parachains/src/hrmp.rs:909:4:
failed to initialize the genesis storage: Module(ModuleError { index: 60, error: [1, 0, 0, 0], message: Some("OpenHrmpChannelInvalidRecipient") })
Error:
   0: Other: wasm call error Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
      WASM backtrace:
      error while executing at wasm backtrace:
          0: 0x708523 - rococo_runtime.wasm!rust_begin_unwind
          1: 0xee0e - rococo_runtime.wasm!core::panicking::panic_fmt::h697e3d119f551db3
          2: 0x6aa03f - rococo_runtime.wasm!polkadot_runtime_parachains::hrmp::initialize_storage::he9bea7bee3f5beff
          3: 0xa2ad3 - rococo_runtime.wasm!<rococo_runtime::RuntimeGenesisConfig as frame_support::traits::hooks::BuildGenesisConfig>::build::h2e20a486c5d31e33
          4: 0x5a8b38 - rococo_runtime.wasm!frame_support::genesis_builder_helper::build_state::h2355dcc9bf4f2368
          5: 0x391273 - rococo_runtime.wasm!GenesisBuilder_build_state

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
2025-01-24 08:37:09.554 DEBUG main wasm-heap: allocator dropped: AllocationStats { bytes_allocated: 41947192, bytes_allocated_peak: 46141184, bytes_allocated_sum: 50439312, address_space_used: 46153272 }

Despite the additional logs, I am still unable to generate the chain specification manually.

After some testing, I noticed that if I commented out the specification of the second channel in the multi_parachains.toml configuration, the command works correctly.
Can this suggest that the issue might be related to how the channels are defined?

[settings]
timeout = 1000
node_verifier = "None"

# Used to start 4 validator nodes and 2 collator nodes - 1 per parachain

[relaychain]
#default_image = "paritypr/polkadot-debug:master"
default_command = "./bin/polkadot"
default_args = [ "-l=parachain=debug,xcm=trace" ]
chain = "rococo-local"

  [[relaychain.nodes]]
  name = "alice"
  validator = true

  [[relaychain.nodes]]
  name = "bob"
  validator = true

  [[relaychain.nodes]]
  name = "charlie"
  validator = true

  [[relaychain.nodes]]
  name = "dave"
  validator = true

[[parachains]]
id = 1000
# Replace this with whichever chain you need to run, e.g. `astar-dev` if you need Astar
chain = "shibuya-dev"
cumulus_based = true

  [[parachains.collators]]
  name = "collator1"
  command = "../../../Astar/target/production/astar-collator"
  rpc_port = 8545
  args = [ "-l=xcm=trace", "--enable-evm-rpc" ]

# For this one you can download or build some other para and run it.
# In this example, `astar-collator` is reused but `shiden-dev` chain is used
[[parachains]]
id = 2007
chain = "shiden-dev"
cumulus_based = true

  [[parachains.collators]]
  name = "collator2"
  command = "../../../Astar/target/production/astar-collator"
  args = [ "-l=xcm=trace", "--enable-evm-rpc" ]

[[hrmp_channels]]
  sender = 2000
  recipient = 2007
  max_capacity = 8
  max_message_size = 8000
  
#[[hrmp_channels]]
#  sender = 2007
#  recipient = 2000
#  max_capacity = 8
#  max_message_size = 8000

I hope this additional information helps.

[Dinonard]

If manual execution failed, then something is wrong with your polkadot binary.
Chain spec construction isn't tied to zombienet in any way.

For the HRMP channel issue, there was (or still is?) a race condition when 2 HRMP clients are defined.
I cannot remember the details, but basically what happened was that a message related to HRMP channel version notification was received before parachains were ready.
I'm not sure whether this was fixed or not.

[Dinonard]

Found the issue: paritytech/polkadot-sdk#1616

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been stalled for 7 days with no activity.