Catch validation error when broadcasting and demote builder

[ltitanb]

Because of optimistic submissions and how we handle simulations, a block may be selected for broadcasting without ever being simulated. Leading to errors like this:

Block accepted but not processed: {"content-type": "application/json", "server": "Lighthouse/v7.0.0-54f7bc5/x86_64-linux", "content-length": "434", "date": "Wed, 08 Oct 2025 19:26:49 GMT"} body: "{\"code\":202,\"message\":\"ACCEPTED: the object was broadcast to the network without being fully imported to the local database: ExecutionPayloadError(RejectedByExecutionEngine { status: Invalid { latest_valid_hash: None, validation_error: Some(\\\"block hash mismatch: want 0x1c4c5c4070f1f84988d207888845dae06a8b5a36a8302a77c011186ae69190b0, got 0x05ed38849ab9048d072c7d4b4ba077a413700feb72b14210bae67bc25b69927c\\\") } })\",\"stacktraces\":[]}" id="27c07520-05e3-4b53-a52a-2d5783cef461"

we should catch this error and demote the builder

[tosynthegeek]

Hello @ltitanb I’d like to take up this issue.

My approach would be to catch this error by checking when the response body contains ExecutionPayloadError or specific messages like "block hash mismatch", and then trigger a demotion for the corresponding builder.

[ltitanb]

Hi @tosynthegeek ! great to hear, the specific error we want to catch right now looks like this:

WARN get_payload: helix_beacon::beacon_client: Block accepted but not processed: {"content-type": "application/json", "server": "Lighthouse/v7.0.0-54f7bc5/x86_64-linux", "content-length": "434", "date": "Wed, 08 Oct 2025 19:26:49 GMT"} body: "{\"code\":202,\"message\":\"ACCEPTED: the object was broadcast to the network without being fully imported to the local database: ExecutionPayloadError(RejectedByExecutionEngine { status: Invalid { latest_valid_hash: None, validation_error: Some(\\\"block hash mismatch: want 0x1c4c5c4070f1f84988d207888845dae06a8b5a36a8302a77c011186ae69190b0, got 0x05ed38849ab9048d072c7d4b4ba077a413700feb72b14210bae67bc25b69927c\\\") } })\",\"stacktraces\":[]}" id="27c07520-05e3-4b53-a52a-2d5783cef461"

this can happen because not all builder blocks are simulated with optimistic submissions. This should trigger a builder demotion indeed

[tosynthegeek]

Hi @tosynthegeek ! great to hear, the specific error we want to catch right now looks like this:

WARN get_payload: helix_beacon::beacon_client: Block accepted but not processed: {"content-type": "application/json", "server": "Lighthouse/v7.0.0-54f7bc5/x86_64-linux", "content-length": "434", "date": "Wed, 08 Oct 2025 19:26:49 GMT"} body: "{\"code\":202,\"message\":\"ACCEPTED: the object was broadcast to the network without being fully imported to the local database: ExecutionPayloadError(RejectedByExecutionEngine { status: Invalid { latest_valid_hash: None, validation_error: Some(\\\"block hash mismatch: want 0x1c4c5c4070f1f84988d207888845dae06a8b5a36a8302a77c011186ae69190b0, got 0x05ed38849ab9048d072c7d4b4ba077a413700feb72b14210bae67bc25b69927c\\\") } })\",\"stacktraces\":[]}" id="27c07520-05e3-4b53-a52a-2d5783cef461"

this can happen because not all builder blocks are simulated with optimistic submissions. This should trigger a builder demotion indeed

Thanks for the clarification @ltitanb. It aligns with my current approach where I’m updating the low-level publish_block function to detect this error and return a BeaconClientError::BlockValidationFailed which in this case I would trigger the demotion.

However, to complete the demotion logic, I need to identify the builder associated with the block. I know the builder’s public key is available in the BidTrace from the auctioneer’s winning bid. Could you clarify how I can access the BidTrace or builder_pubkey in the context of ProposerApi::_get_payload?

[ltitanb]

not currently, you can probably add it to GetPayloadResultData

[tosynthegeek]

not currently, you can probably add it to GetPayloadResultData

@ltitanb made a PR #315 for this