[Node Operator Question] What could cause long re-sync periods after resetting a node?

[hazim-j]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

what conditions can cause op-node to emit the log Walking back L1Block by hash?

For more context when I restart an op-node I'm getting multiple of these logs:

t=2024-02-01T03:09:03+0000 lvl=info msg="Walking back L1Block by hash"        curr=0x...:19100499 next=0x...:19100498 l2block=0x...:9800354

This will repeat continuously until I get something like this (log hex strings trimmed):

t=2024-02-01T05:07:00+0000 lvl=info msg="Found next batch"                       batch_type=SingularBatch batch_timestamp=1,704,113,197 parent_hash=0x... batch_epoch=0x...:18912654 txs=25
t=2024-02-01T05:07:00+0000 lvl=info msg="generated attributes in payload queue"  txs=26 timestamp=1,704,113,197
t=2024-02-01T05:07:00+0000 lvl=info msg="Sync progress"                          reason="reconciled with L1"            l2_finalized=0x...:8661739 l2_safe=0x...:8661925 l2_safe_pending=0x...:8661925 l2_unsafe=0x...:9983414 l2_engineSyncTarget=0x...:9983414 l2_time=1,706,756,175 l1_derived=0x..:18912670
t=2024-02-01T05:07:00+0000 lvl=info msg="Advancing bq origin"                    origin=0x...:18912671 originBehind=false
t=2024-02-01T05:07:00+0000 lvl=info msg="Advancing bq origin"                    origin=0x...:18912672 originBehind=false
t=2024-02-01T05:07:00+0000 lvl=info msg="Advancing bq origin"                    origin=0x...:18912673 originBehind=false

While op-node is verifying op-geth is blocked. It's taking several hours to resync a node after a reset. I initially found the problem on a Base node but seems like its also happening on Optimism too.

I'm making sure to shut down the node gracefully (i.e. op-node first then op-geth, allowing sufficient timeout for op-geth to write to disk).

Once sync'd there is no issues keeping up with the tip so hardware must be okay.

Running op-node with following params:

--l1=$RPC
--l2=http://localhost:8551
--network=$NETWORK
--rpc.addr=127.0.0.1
--rpc.port=9545
--l2.jwt-secret=/secrets/jwt.hex
--rollup.halt=minor
--rollup.load-protocol-versions=true

Protocol Description

• op-node v1.4.0
• op-geth v1.101305.1
• network name: op-mainnet (experienced the same issue with base as well)
• chainId: 10
• L1 cahin: Ethereum mainnet

Node Logs

No response

Additional Information

No response

Feedback

No response

[sbvegan]

Hey @hazim-j,

The reason for the Walking back L1Block by hash message is because its walking back to the last finalized block to begin resyncing from there. This is because the last finalized block is the last block that was marked as derived by the L1. Here's information about unsafe, safe, and finalized tags.

The reason for the long re-sync period, is for some currently unknown reason, your node isn't marking the latest finalized block. So it'll keep walking back until it finds its known last finalized block or all the way back to genesis.

To find out your unsafe, safe, and finalized blocks, you can use the following RPC requests:

Last Finalized Block

curl $RPC_URL \
-X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["finalized", false],"id":0}'

Latest block (unsafe)

curl $RPC_URL \
-X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest", false],"id":0}'

Latest safe block

curl $RPC_URL \
-X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["safe", false],"id":0}'

Additional Comments

Your node versions are not the latest minimum releases. You can see that here: https://docs.optimism.io/builders/node-operators/releases

[hazim-j]

Dumping more info on one of our nodes:

• Finalized block: 113905756 (~40 days ago)
• Unsafe block: 115630075 (~20 hours ago when the node was restarted)
• Safe block: 113905756 (~40 days ago, same as finalised)

Again, I'm not sure why our nodes aren't finalising blocks 🤔

[sbvegan]

@hazim-j can you try running the latest required node software and let me know if the blocks start getting marked as finalized?

[trianglesphere]

@hazim-j what are the machine resources for op-geth (CPU, RAM, Disk Size + Type). Also what is your L1 RPC url like (your own node or a RPC provider)

[hazim-j]

@sbvegan Running the latest required node software has the same issue.
@trianglesphere 8vCPU, 32GB RAM, 4TB SSD. Once synced the node has no problem staying sync'd. The L1 RPC is also our own node.

Also sharing this related issue on infinite walk back loop if the L1 gets rate limited: ethereum-optimism/optimism#9347

Although still not sure how the node could go for weeks without finalising.

[trianglesphere]

@hazim-j please upgrade to op-node 1.5.1. If you pull the log we could potentially pin it to a known bug in that version, but the behavior you are experience is known to be related. Unfortunately that finalization process will always take a long time (unless you manually finalize)