[Node Operator Question] <Block synchronization speed issues>

[junghwan-chon]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

We are currently operating an archive node, and block synchronization was initially completed, but now block synchronization is becoming more and more delayed.
Could you possibly know the cause?

INFO [01-05|17:56:31.958] Chain head was updated number=114,417,083 hash=a885d3..70defa root=a371bd..938f1a elapsed="246.213µs" age=3h20m48s
INFO [01-05|17:56:33.087] Imported new potential chain segment number=114,417,084 hash=35ae59..177fc5 blocks=1 txs=15 mgas=4.930 elapsed=1.122s mgasps=4.394 age=3h20m48s triedirty=0.00B
INFO [01-05|17:56:33.089] Chain head was updated number=114,417,084 hash=35ae59..177fc5 root=83e67d..f86649 elapsed="484.608µs" age=3h20m48s

Yesterday there was about an hour difference, but now there is a difference of 3 hours and 20 minutes.
Node specs are 8 core 64gb.

Protocol Description

op-geth v1.101304.2
op-node v1.4.0
chainId: 10
network: 10

Node Logs

No response

Additional Information

No response

Feedback

No response

[opfocus]

Have you observed the read/write speed of your hard disk and the status of your internet connection?

[sbvegan]

Hey @junghwan-chon, can you tell us about what your L1 rpc provider setup looks like and the rpc.kind flag settings? Can you provide all of your op-node/op-geth configurations for reference?

[sbvegan]

What Ethereum L1 provider are you using or what kind of node are you running (i.e. geth, erigon, etc)?

[junghwan-chon]

I'm currently using geth.

[sbvegan]

Sorry for letting this go stale, if you're still experience lag. I wonder if you missed a network upgrade and the sync speed issue is actually because your blockchain diverged. Then its just relying on L1 data for the source of truth, so it lags behind the L2 unsafe head.

[cromaniuc]

@sbvegan I'm having exactly the same issue after upgrading to:

  OP_GETH_CHAIN_VERSION: v1.101305.1
  OP_NODE_CHAIN_VERSION: v1.4.3

I wonder if you missed a network upgrade

Is there a way to check your assumption? Thanks and much appreciated!

[sbvegan]

Yeah you can check the blocks that you're producing match up with the blocks known to be correct i.e. blocks reported to https://optimistic.etherscan.io/. My hunch was that you missed a network upgrade and because the consensus rules were updated, your chain diverged and is now producing nonsense blocks. But the rollup software always uses the L1 as the canonical source of truth so it updates its fork when it derives from the L1 and that might be the source of the lag.

[moorow]

I'm also struggling with this, could someone explain where the bottlenecks are?

You can safely assume that my disk I/O and network bandwidth are not bottlenecks.

How much of an effect does the choice of L1 have on the Sync? For example Alchemy vs ETH node running on same machine?

Does the ETH node need to be Archive or is a Full node sufficient?

What is the fastest possible configuration for synchronising op-geth?

[sbvegan]

could someone explain where the bottlenecks are?

The bottlenecks are the block receipt fetching and validation. So you can optimize that by setting your l1.rpckind. If you trust your L1 node, you can set your l1.trustrpc to true.

How much of an effect does the choice of L1 have on the Sync? For example Alchemy vs ETH node running on same machine?

We don't currently have benchmark comparisions

Does the ETH node need to be Archive or is a Full node sufficient?

It depends on your use case. Archive nodes are for historical executions and full blockchain data.

What is the fastest possible configuration for synchronising op-geth?

Setting the configurations I mentioned above

[cromaniuc]

Using --l1.rpckind=erigon and --l1.trustrpc for an archive node (I'm running it since couple of months ago without any issues). With the latest 2 versions the node is synching very slow. It takes 3-4 hours to catch up with the tip, after a simple restart.

Any other hints will be highly appreciated! Thanks!

[sbvegan]

This is from the release notes in op-node v1.4.3:

This is a strongly recommended update of op-node. It brings back an important receipts validation check to the op-node, which was disabled since December '23 when the trustrpc option was set to true. There have been numerous reports of L1 rpc endpoints returning incomplete receipt lists, which could lead to localized chain splits without receipts validation. With this release, trustrpc can again safely be set to true.

Maybe having --l1.trustrpc set to true was causing a chain divergence. Its worth also bumping your nodes to the latest required version to prepare for the next network upgrade: https://docs.optimism.io/builders/node-operators/releases#required-version-by-network