[Node Operator Question] state root diverging

[aymentrosh]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

we are trying to upgrade op-node to the latest release for the Pectra upgrade we have some custom changes on op-node and op-batcher after rebasing and using the updated op-node, op-batcher, and op-geth we tested, we see that it's giving different state roots from the old running l2

Steps to Reproduce

cat test.bash
number=$(curl --silent localhost:8546 \
  -X POST \
  -H "Content-Type: application/json" \
  --data '{
  "method": "eth_getBlockByNumber",
  "params": [
    "latest",
    false
  ],
  "id": 1,
  "jsonrpc": "2.0"
}' | jq .result.number)

echo "Replica at block "$number
nn=$(echo $number| sed s/0x// | tr -d "\"")
BLOCK_NUMBER=$((16#$nn))
echo $BLOCK_NUMBER
curl --silent localhost:8546 -X POST   -H "Content-Type: application/json"   --data '{  "method": "eth_getBlockByNumber",  "params": ['$number',false  ],  "id": 1,  "jsonrpc": "2.0"}' | jq .result.stateRoot


curl --silent $mainnet-RPC-URL   -X POST   -H "Content-Type: application/json"   --data '{  "method": "eth_getBlockByNumber",  "params": ['$number',false  ],  "id": 1,  "jsonrpc": "2.0"}' | jq .result.stateRoot

Which component is affected?

• op-geth
• op-node
• Other

Expected vs. Actual Behavior

we expect to have the same state root

Replica at block "0x1"
1
"0x5a7c871d8390e41d50026a3edf26cc93ab3070b722836aec390d26a95158af75"
"0x8250956be3f2764525cd99332785f94fd28863d29774a7a7895c6adfe91d37eb"

Protocol Information

this is mainnet, and we are targeting op-node/v1.12.2 and op-geth/v1.101503.1

[opfocus]

the old one and the new one are op-mainnet nodes right?

[opfocus]

So what you are saying is that you run your own L2, and have a replica node of your own.
Replica at block "0x1"
This may mean there is a problem with replia synchronization. Can you check why 1 is returned here?

[aymentrosh]

In that time it's not synced yet even if i try a different block it keeps returning different state roots

[opfocus]

If your sequencer node is running normally, then we have time to investigate the issues with the replica node. However, if the sequencer node experiences a failure, it may lead to persistent inconsistencies — for example, if the sequencer is down for too long, the replica node may start deriving new blocks from L1 instead of receiving them from the sequencer.

[opfocus]

Could you provide the logs of the op-node of the sequencer node and the replica node?
By the way, this might be unrelated, but here’s a link that could be useful: https://docs.optimism.io/operators/chain-operators/tools/op-conductor

[jsvisa]

@aymentrosh correct me if I'm wrong, you're running two L2 op-nodes, let's say one is the primary node, and the other one is the replica node. And now the state root of the same block of the primary and replica nodes are different? Besides of the state root, does the other fields(eg: block hash) are the same or not?