Unknown CBOR structure

[liv-io]

Hello,

Since the Chang hard fork (preprod) on Saturday, August 24th, around midnight UTC, the Carp process has been repeatedly crashing. The logs show the following error message:

2024-08-26T09:45:22.399992Z ERROR oura::sources::n2c::run: N2C error: Unknown CBOR structure: 820785828a1a0027fb821a04196a0d582036f5b4a370c22fd4a5c870248f26ac72c0ac0ecc34a42e28ced1a4e15136efa45820087932a594533b96a2abd339715dd222835d36856ac2c0b8fcd171e3ec944b8e5820b5b14e8eb25a9366e0c1a43afcd8d0859249202f3a8dc03b13f416c70d4de7d7825840358661d42ed3ceb1fbcb07d773d3472f1d68842370af0034c3fe289653782303dc3950da41db8038d8bb8c242df8a574f7a28cd67922a5d33e1a74126a71542a5850c0b6fb4330f571d08672557969b83566bc8133f8988bbe1bfc7a8fc4e4322cfd7f9ab6d1dce5aa2fc521715153fe29b24d1a5507c2941bd48c87cbb4b2a01dabea07898f1a09d15628b91f0560a5f00419057558203489d0231efe72dc60405458891207f7b7a90a4cb9e395eabd8a53ea6ac856c28458208cc744c58c7a05709e02beb11b63c311687a1c1308b11e483d1c728f4f6fbf7d041901fd5840ba3364505403cd1f409595472bbf1e9e020fa331b39cf9c8a9652841ad6fe609de243d2368861351e83c685eff105f8b7bb7389d11c898cb94876202803115008209015901c0e8adde62449b3410a78a4299e695bbb3e29c4114c025450bbfbf461ba5ead9d54c20c61097ed72d7f1b6c1613364af7bcb02127c3fd1a9dab21a84414331a1043ff0a81985e24e43024713526d2647646c5cc5967ee94d551d14a0839ecc17029cd83dde86a4c8ff63b70bae03d474a99ca960b4f540f499720788af1165043a357e395833dd8dd5d1da93f45682643d38f7eb45b3100d8e8ee1e87c648eb042ade8c809ad46b8df7c21996be957b140582ca76d92a7e69a98a776a6fdf5c8512d6d1875efdd12da5187b565a4cf3fa2b60c462038fad4d93a3171b15207051fc4e83d71f4506cc1a500fc0c9f8937041f379dd0e5f54030aaf910dfeaaa9d650b2032913db7835abc83e5ef6c4cb8b1551e39582d8f846cb29a65a2e5a1209bac3a6ef8ca66be887138cc8b8e018eed01668d41f46e0b667ef77c64d5134f0ce9e8bab5cfe8aad751b83770213ed40d2304aeec615b903fb0a45f7151b5c2a32a525e8be894cf798bf437b7279e0650e197b1bd0ad3c58194719d9beab06a5f968e917cc68a627df194e19d08bfc18482638deb4f2da42f7096d2cbc7e8b4f472f1027f53cca668252e30946f6e9ca4c3e224cd40f764ea59fe85d420b0bbe782a50081825820b151981e025e3b4593640adf888b9caf14eeeeaec8b7ac0e06e3b8347562ca3000018182581d6038be2f76b5bc38cb9efca623097b278e4a56b7dcf61ed7555c2c60221b000000019980b57e021a0002a179031a04196c16075820bdec441db41221fbc91d80069f2ec6921c38406f4bd593f34b3fe262208ffb66a40081825820685574e6bf7e112635fb033730584603f23ed4c3d4dfdbc937ea64dfd69b9ee0000181825839007452a081bd1123e2aeb874a7d4e2f3bd65e54ea22f60fe3a73af411eab2a3abab6339252c9fdb9a51ae58099654e2391ce07616f0e1b734b821a79b5cd5aa1581c70f213383c0ac04d9ac817264c19eff31234d88eaba5d66fbaa332cda14441584f5301021a00032ec1075820800ef669e83c5a9959f33a4d08d6134e0e40c3448c780cec35d19f14cba21c1082a10081825820f6641bdf366ff90a3592220e88aef7859ebd5ad68a614418dc66b144b23ff83b5840076f5e921bf1299ae09cf0909a12afbb8f732d55e399b77faac0a8b4faab1d2f728d5153b0e88aa2737faaa275f4e18fe42cd12d588ea6690a083db562e17c03a100818258208af3acd3b7774ba6f20be0ffe102f49a9c9a6d9ec1ff8f60103835744abf5c615840c36e3fa20584b8b1cb740087186f24704c8b1a83bbfe4fdc0571a80fd2d5dffa5b58366a49634c39381faaf554f30838ef6a3cea040c29c634153d36d5941c0fa200d90103a100a101a26974696d657374616d706a313732343435373539346d6162736f6c7574655f736c6f7468363837373433393401a1192b69a8636b65798378406135303130313032353831646530616232613361626162363333393235326339666462396135316165353830393936353465323339316365303736313666306578403162373334623033323732303036323135383230373532633765393333383032323566623564643835363366666339356464383466653066663638396463353174663134633531333135626232666666303533633164747970656b434f4d4d49544d454e545366666f726d6174664349502d3330677061796c6f6164a464747970656b434f4d4d49544d454e54536b636f6d6d69746d656e7473a1781843465f53554d4d49545f323032345f31304243432d444556a164686173687840663866613063386565353466663663326330366638353533643462333031366636313736316631333336653636613464396136656263343738333831633830366c6372656174696f6e536c6f741a041969e46d736368656d6156657273696f6e65312e302e306868617368547970656b424c414b4532425f323234697369676e617475726586784038343538346161333031323730343538316465306162326133616261623633333932353263396664623961353161653538303939363534653233393163653037784036313666306531623733346236373631363436343732363537333733353831646530616232613361626162363333393235326339666462396135316165353830784039393635346532333931636530373631366630653162373334626131363636383631373336383635363466343538316334396534363930643633643566316539784035396164323933383638663835313937353830653262376665373630343839306364346463383036353834306363323962646564343030393535323965626135784032366334353134346132343962306535303635333333646536636162623737353734316332346133366531343833326261333632303131643634393530333735782c333663646230633564613833373965346263336432623938383738626332373332323430623734353532306669737562466f726d61746443424f526d7369676e61747572655479706569484153485f4f4e4c5980    
2024-08-26T09:45:22.400014Z  WARN oura::sources::n2c::run: unrecoverable error performing chainsync, will exit    
2024-08-26T09:45:22.600449Z ERROR carp: Processing loop finished with error, stopping engine error=Can't fetch oura event: RecvError
2024-08-26T09:45:22.600594Z  INFO carp: Engine is stopped successfully

Resynchronizing Carp from scratch, including completely dropping the PostgreSQL database, did not resolve the issue for me.

Setup

Network

PreProd

Specifications

The hardware specifications are:

• 4 cores
• 10 GB
• 4 NVMe drives in RAID-5

Versions

The software versions are:

• Debian 12
• cardano-node 9.1.0
• postgresql 15.x
• carp 3.2.0

Configuration

• default.yml

  source:
    type: oura
    socket: "/var/local/cardano-node/db/node.socket"
    bearer: Unix # Unix
  sink:
    type: cardano
    db:
      type: postgres
      database_url: postgresql://carp:<password>@127.0.0.1:5432/carp
    network: preprod # preview / preprod / testnet
  start_block:
  

Please let me know if I need to provide further details about our setup.

Thank you for your help.

[rooooooooob]

I'm going to bump oura and see if this fixes it. I was already able to sync preprod using cardano_net without issues.

Is it possible for you to use cardano_net instead of oura? That way you can avoid oura and have it go directly through CML instead which can parse everything fine.

I tried running the "Unknown" block cbor reported there through CML directly and it works in 6.0.0 / 5.3.1 / 5.2.0 (I didn't test any older). Generally CML has far fewer parsing problems since the serialization/deserialization part is generated directly from the conway.cddl file so human error is minimized.

[liv-io]

After changing the source.type from oura to cardano_net, I ran into a new error when synchronizing from scratch:

2024-08-29T11:30:25.620308Z  INFO carp::sinks::cardano: Starting epoch 50 at block #598073 (268f8210f355e9eb2a2f4a22a774cf6c85f48cca768c69d4ffcab07926716b1b)
2024-08-29T11:31:11.678262Z  INFO carp::sinks::cardano: Finished processing epoch 50 after 43s (+2s)
2024-08-29T11:31:11.678278Z  INFO carp::sinks::cardano: Starting epoch 51 at block #616080 (9ecca5c54559257ca8bf229a263213077d760b53d495e3d1befdde768bb67f48)
2024-08-29T11:31:53.195443Z  INFO carp::sinks::cardano: Finished processing epoch 51 after 39s (+1s)
2024-08-29T11:31:53.195460Z  INFO carp::sinks::cardano: Starting epoch 52 at block #633872 (b8e08fa5756a2dbbfa9a0d63ad650e44cffeb8eea2121b230f9e81da28751d51)
2024-08-29T11:32:59.508879Z ERROR carp: Processing loop finished with error, stopping engine error=Error while interacting with the Persistent storage of the Multiverse

The new configuration file is:

source:
  type: cardano_net
  relay:
    - 127.0.0.1
    - 30000
sink:
  type: cardano
  db:
    type: postgres
    database_url: postgresql://carp:<password>@127.0.0.1:5432/carp
  network: preprod # preview / preprod / testnet
start_block:

Can you spot any obvious mistakes in my setup?

Thank you for your help.

[rooooooooob]

I don't think so. I ran it locally with this:

source:
  type: cardano_net
  relay:
    - localhost
    - 3001

sink:
  type: cardano
  db:
    type: postgres
    database_url: postgresql://carp:<password>@localhost:5432/carp_preprod
  network: preprod # preview / preprod / testnet / custom

start_block:

but it looks identical just with localhost instead of 127.0.0.1 but that just resolves to the same thing in the end. How are you running your cardano node? I was running it from the docker image on 9.0.0. What version is your node on? Were you re-syncing the DB from scratch or did it have your old blocks in the carp DB from when you were running from oura?

[rooooooooob]

Also what version of carp are you using? Does it still happen with #198?

[liv-io]

Thanks for your efforts.

I was running it from the docker image on 9.0.0. What version is your node on?

I am running cardano-node 9.1.0

Also what version of carp are you using?

I am running carp 3.2.0 as stated in the description.

Does it still happen with #198?

Do you want me to compile and run carp from this branch?

[liv-io]

@rooooooooob
I ran into the same error after updating:

• cardano-node from 9.1.0 to 9.1.1
• carp from 3.2.0 to 3.3.0

I've purged the PostgreSQL and resynced carp (PreProd) from scratch. The error message is:

2024-09-16T08:54:40.967532Z  INFO carp::sinks::cardano: Starting epoch 51 at block #616080 (9ecca5c54559257ca8bf229a263213077d760b53d495e3d1befdde768bb67f48)
2024-09-16T08:55:22.104724Z  INFO carp::sinks::cardano: Finished processing epoch 51 after 39s (+1s)
2024-09-16T08:55:22.104748Z  INFO carp::sinks::cardano: Starting epoch 52 at block #633872 (b8e08fa5756a2dbbfa9a0d63ad650e44cffeb8eea2121b230f9e81da28751d51)
2024-09-16T08:56:27.808463Z ERROR carp: Processing loop finished with error, stopping engine error=Error while interacting with the Persistent storage of the Multiverse

Please let me know if I can test anything with a development branch/release. Thanks for your help.

[rooooooooob]

@ecioppettini any idea what could be causing this? Error while interacting with the Persistent storage of the Multiverse doesn't sound very descriptive.