feat: messaging precompiles

[evilrobot-01]

An adaptation of #528 to use precompiles for ISMP/XCM messaging.

NOTE: there is currently a single failing test, which is an integrity test added to the messaging pallet. It effectively highlights the fact that a max encoded XCM response is potentially prohibitively large. Most response are probably tiny in comparison, so a more practical approach might be to measure the largest xcm responses in practice and size accordingly. I have intentionally left this as a failing test as I feel it needs further consideration.

ISMP

The following diagram illustrates the flow of the ISMP implementation:

flowchart TB
    A["Contract"] -- Ismp::get/post --> B["messaging::ISMP Precompile"]
    B -- transports::ismp::get/post --> n4["Take Deposit"]
    n1["runtime::Router"] -- ISMPModule::on_response --> n2["messaging::ISMP Module"]
    n2 --> n3["Callback?"]
    D["Relayer"] -- "pallet-ismp::handle(messages)" --> n1
    n3 -- "<span style=padding-left:><span style=padding-left:>onGetResponse/onPostReponse</span></span>" --> A
    C["pallet-ismp::ISMP Dispatcher"] -.- D
    n4 -- IsmpDispatcher::dispatch_request --> C
    C --> n5["Persist Request/Message"]

    n4@{ shape: rounded}
    n1@{ shape: rect}
    n2@{ shape: rect}
    n3@{ shape: diam}
    D@{ shape: hex}
    n5@{ shape: rounded}

Loading

XCM

The following diagram illustrates the flow of the XCM implementation:

flowchart TB
    A["Contract"] -- Xcm::newQuery --> B["messaging::XCM Precompile"]
    B -- transports::xcm::new_query --> n4["Take Deposit"]
    n1["xcm_executor"] -- Pallet::xcm_response --> n2["messaging::Pallet"]
    n2 --> n3["Callback?"]
    D["XCM"] -- </br> --> n1
    n3 -- onQueryResponse --> A
    C["pallet-xcm"] -.- D
    n4 -- Xcm::new_notify_query --> C
    C --> n5["Persist Request/Message"]
    A -- Xcm::send --> B
    B -- Pallet::send --> C

    n4@{ shape: rounded}
    n1@{ shape: rect}
    n2@{ shape: rect}
    n3@{ shape: diam}
    D@{ shape: hex}
    n5@{ shape: rounded}

Loading

Outstanding Items

The following items remain as outstanding:

• ensure callback has weight specified: the implementation of the new_query helpers in the ink api crate, as well as the precompile itself, should ensure that if a callback is specified, it specifies non-zero weight. This is just a simple check to avoid callbacks failing later, but is no means sufficiently robust as the developer will always need to ensure that sufficient weight is specified.
• helper functions for converting from bytes to an expected ismp/xcm response type: a callback for an ISMP get is strongly typed as Vec<StorageValue> but a call to get_response simply returns an opaque blob of bytes.
• get_response to include encoding specification: persisted responses are scale-encoded, meaning that if a contract uses get_reponse for an ISMP response, it will be scale-encoded bytes. This differs to the callback, which would be encoded as per the callback specification. Adding an encoding field to the get_response function should be sufficient to resolve this.
• top up callback weight + tests: the top_up_callback_weight functionality was not ported due to time constraints. It should be an easy add to the existing precompile(s). Additional consideration should be given to the storage deposit limit which was omitted in the previous implementation.
• consider implementing weigher function: https://github.com/paritytech/polkadot-sdk/blob/d9f451a6b94ab2cf39371ee5192130379eb6e199/polkadot/xcm/pallet-xcm/src/precompiles/IXcm.sol#L40

[sc-3823]

[codecov-commenter]

Codecov Report

❌ Patch coverage is 88.21805% with 402 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.03%. Comparing base (181d1ef) to head (91b5312).

Files with missing lines	Patch %	Lines
...ets/api-vnext/src/messaging/precompiles/ismp/v0.rs	85.08%	52 Missing and 39 partials ⚠️
...lets/api-vnext/src/messaging/precompiles/xcm/v0.rs	87.82%	45 Missing and 25 partials ⚠️
pallets/api-vnext/src/messaging/weights.rs	50.00%	65 Missing ⚠️
pallets/api-vnext/src/messaging/transports/ismp.rs	92.95%	22 Missing and 14 partials ⚠️
runtime/devnet/src/config/api/mod.rs	0.00%	33 Missing ⚠️
pallets/api-vnext/src/messaging/precompiles/v0.rs	87.86%	19 Missing and 10 partials ⚠️
pallets/api-vnext/src/messaging.rs	93.39%	15 Missing and 6 partials ⚠️
pallets/api-vnext/src/messaging/tests.rs	96.72%	12 Missing and 6 partials ⚠️
runtime/devnet/src/config/ismp.rs	0.00%	11 Missing ⚠️
runtime/devnet/src/lib.rs	0.00%	10 Missing ⚠️
... and 4 more

@@                    Coverage Diff                    @@
##           fungibles-precompiles     #584      +/-   ##
=========================================================
+ Coverage                  69.47%   72.03%   +2.56%     
=========================================================
  Files                        134      144      +10     
  Lines                      21538    24940    +3402     
  Branches                   21538    24940    +3402     
=========================================================
+ Hits                       14963    17966    +3003     
- Misses                      6243     6541     +298     
- Partials                     332      433     +101     

Files with missing lines	Coverage Δ
pallets/api-vnext/src/fungibles/precompiles/v0.rs	88.27% <ø> (+0.09%)	⬆️
pallets/api-vnext/src/messaging/deposits.rs	100.00% <100.00%> (ø)
runtime/devnet/src/config/contracts.rs	89.04% <ø> (ø)
pallets/api-vnext/src/lib.rs	95.79% <80.00%> (+0.07%)	⬆️
pallets/api-vnext/src/messaging/benchmarking.rs	98.41% <98.41%> (ø)
pallets/api-vnext/src/mock.rs	89.69% <93.26%> (+4.84%)	⬆️
pallets/api-vnext/src/messaging/transports/xcm.rs	95.96% <95.96%> (ø)
runtime/devnet/src/lib.rs	4.12% <0.00%> (-0.09%)	⬇️
runtime/devnet/src/config/ismp.rs	0.00% <0.00%> (ø)
pallets/api-vnext/src/messaging/tests.rs	96.72% <96.72%> (ø)
... and 7 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Daanvdplas]

Damn, it really is a super cool feature. Well done on the conversion, I see you made several improvements along the way which were remaining todos, amazing!

I have not been able to dive into the details, as well as benchmarks, but the overall features and integration tests look very good. Next week I will make this my priority.

Curious, would it ever be possible to return the result of the destination chain when doing an xcm send?

And are there still big open standing issues unsolved, I remember something along the lines of child tries for ismp benchmarking (?). This is not a problem but I would like to have a look at this for the next review to get a better understanding.

The precompiles and contracts look super clean, that with the complexity of these protocols, incredible!

[evilrobot-01]

Curious, would it ever be possible to return the result of the destination chain when doing an xcm send?

Thats what the example already shows?

pop-node/pop-api-vnext/examples/messaging/lib.rs

Lines 97 to 133 in 76e8e85

	#[ink(message, payable)]
	pub fn transact(&mut self, call: Vec<u8>, weight: Weight) {
	let dest = Location::new(1, Parachain(self.para));
	let call = DoubleEncoded::<()>::decode_all(&mut &call[..]).unwrap();
	// Register a new query for receiving a response, used to report transact status.
	let (id, query_id) = api::xcm::new_query(
	VersionedLocation::V5(dest.clone()),
	self.env().block_number().saturating_add(100),
	Some(Callback::new(
	self.env().address(),
	ENCODING,
	0x641b0b03,
	Weight::from_parts(800_000_000, 500_000),
	)),
	);
	// TODO: provide an api function for determining the local para id and max weight value
	// for processing the reported transact status on the local chain.
	let response = QueryResponseInfo {
	// Route back to this parachain.
	destination: Location::new(1, Parachain(4_001)),
	query_id,
	max_weight: Weight::from_parts(1_000_000, 5_000),
	};
	// Send transact message.
	let fees: Asset =
	(Location::parent(), u128::try_from(self.env().transferred_value()).unwrap())
	.into();
	let message: Xcm<()> = self._transact(call, weight, fees, response);
	let mut hash = [0u8; 32];
	Blake2x256::hash(&message.encode(), &mut hash);
	let result = api::xcm::send(dest.into_versioned(), VersionedXcm::V5(message));
	self.env().emit_event(XcmRequested { id, query_id, hash });
	}

The _transact function adds an instruction to report the transact status back:

pop-node/pop-api-vnext/examples/messaging/lib.rs

Lines 145 to 170 in 76e8e85

	fn _transact(
	&self,
	call: DoubleEncoded<()>,
	weight: Weight,
	fees: Asset,
	response: QueryResponseInfo,
	) -> Xcm<()> {
	let beneficiary = hashed_account(4_001, self.env().account_id()); // todo: para id getter
	Xcm::builder_unsafe()
	.withdraw_asset(fees.clone())
	.buy_execution(fees, WeightLimit::Unlimited)
	.set_appendix(
	Xcm::builder_unsafe()
	.refund_surplus()
	.deposit_asset(
	All,
	Location::new(0, AccountId32 { network: None, id: beneficiary.0 }),
	)
	.build(),
	)
	.set_error_handler(Xcm::builder_unsafe().report_error(response.clone()).build())
	.transact(OriginKind::SovereignAccount, weight, call)
	.report_transact_status(response)
	.build()
	}
	}

And are there still big open standing issues unsolved, I remember something along the lines of child tries for ismp benchmarking (?). This is not a problem but I would like to have a look at this for the next review to get a better understanding.

The ismp child trie benchmarking has been overcome using the same approach as pallet-contracts/revive:

pop-node/pallets/api-vnext/src/messaging/benchmarking.rs

Lines 132 to 137 in 76e8e85

	// IMPORTANT NOTE: `skip_meta` and `pov_mode = Measured` currently used due to an issue with the
	// usage of the `RequestCommitments` storage item within `pallet_ismp`'s child trie, which
	// results in massive proof size values resulting from complexity parameters, preventing
	// compilation of the resulting weights. Reducing max limits on pallet config also avoids the
	// issue, but results in values which are too small to be useful. See `pallet-contracts` and
	// `pallet-revive` benchmarks for similar usage with contract child tries.

[Daanvdplas]

Could you explain why you set those values here?

[evilrobot-01]

The contract must effectively specify the values required as limits when the response is dispatched back to the contract via

pop-node/runtime/devnet/src/config/api/mod.rs

Line 284 in c08c5b9

let mut output = Revive::bare_call(

.

The cost of the gas_limit specified at the time of request is reserved, with any unused weight/cost being refunded after callback execution. The storage deposit limit covers the max amount that should be reserved for any state that the receiving contract creates. Whilst this is typically the same contract, there is now the dest field on the callback which allows specification of a contract address other than the caller for a more powerful implementation.