[Node Operator Question] Node consistently lagging behind by 2 hours

[J1a-wei]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

Issue

Hi team, my node is unable to catch up with the latest block, consistently lagging behind by 2 hours. I suspect it's likely due to a P2P issue. Does GETH require exposing its own P2P for data synchronization?

Thinking

Our deployment environment is within k8s, with connectivity to the internet, but via NAT translation. Therefore, I suspect the issue is due to insufficient peers.

I attempted to call the geth peer RPC, which showed 0 peers. After consulting some documentation, it seems that the synchronization of OP-geth does not rely on GETH's P2P, is that correct? It appears to entirely rely on the OP-node's P2P.

I tried calling the RPC of opnode. Result 9 peers but found no inbound peers.

/ # curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"opp2p_peers","params":[true],"id":1}'  http://localhost:9545
{"jsonrpc":"2.0","id":1,"result":{"totalConnected":9,"peers":{"16Uiu2HAm1AdeLrPrnzUAzKuF1EYiFMCg7Q3HJFJy1wE24f4aAfWi":{"peerID":"16Uiu2HAm1AdeLrPrnzUAzKuF1EYiFMCg7Q3HJFJy1wE24f4aAfWi","nodeID":"cadc518ae2deea2b73277bac45ddc51840eaa0b09638bf2490e23140cbe41c42","userAgent":"optimism","protocolVersion":"","ENR":"enr:-Jy4QLiTOo0OARxAqZI_Fyy-S9jxyGNd6Eldwp_OV2hOnQ7vE6-zDXZk2e0Kd-2Hm1lZ2u9P0MWjMct_dRb_VnCxWwKGAYwIb3t_gmlkgnY0gmlwhLAJFweHb3BzdGFja4IKAIlzZWNwMjU2azGhAlVLxNIX1NzdbbV0uj3s_jwolJsd-llOf2vB95p8Fu1rg3RjcIIkBoN1ZHCCxAQ","addresses":["/ip4/176.9.23.7/tcp/9222/p2p/16Uiu2HAm1AdeLrPrnzUAzKuF1EYiFMCg7Q3HJFJy1wE24f4aAfWi"],"protocols":["/libp2p/autonat/1.0.0","/meshsub/1.0.0","/meshsub/1.1.0","/opstack/req/payload_by_number/10/0","/floodsub/1.0.0","/ipfs/id/1.0.0","/ipfs/id/push/1.0.0","/ipfs/ping/1.0.0"],"connectedness":1,"direction":2,"protected":false,"chainID":10,"latency":330652315,"gossipBlocks":true,"scores":{"gossip":{"total":-5.04,"blocks":{"timeInMesh":0,"firstMessageDeliveries":0,"meshMessageDeliveries":0,"invalidMessageDeliveries":0},"IPColocationFactor":0,"behavioralPenalty":0},"reqResp":{"validResponses":0,"errorResponses":0,"rejectedPayloads":0}}},"16Uiu2HAm82SfS2ivT45NnzhxomV8U2MSbDvpVUBJpMNudPJGSVut":{"peerID":"16Uiu2HAm82SfS2ivT45NnzhxomV8U2MSbDvpVUBJpMNudPJGSVut","nodeID":"5f2c26fe91b8e88c53d9bb6a38e89dd4230a94cabdb0acd787f9b69e58fff9c4","userAgent":"optimism","protocolVersion":"","ENR":"enr:-Jy4QFr0ERtfgcxt3IB_atcbEJB-qVPT4i-FdyCaSEB-eGV5AfbJucAKG3jOzFjC5G0InfR0Eclw1-WxJjZXj7eTEFuGAY2I05-HgmlkgnY0gmlwhA_rCpeHb3BzdGFja4IKAIlzZWNwMjU2azGhArs0g6utGGmO0ysq92gymRwcXTFFpZcNP13dQlTSE87jg3RjcIIkBoN1ZHCCkBo","addresses":["/ip4/15.235.10.151/tcp/9222/p2p/16Uiu2HAm82SfS2ivT45NnzhxomV8U2MSbDvpVUBJpMNudPJGSVut"],"protocols":["/ipfs/id/push/1.0.0","/ipfs/ping/1.0.0","/libp2p/autonat/1.0.0","/meshsub/1.0.0","/meshsub/1.1.0","/opstack/req/payload_by_number/10/0","/floodsub/1.0.0","/ipfs/id/1.0.0"],"connectedness":1,"direction":2,"protected":false,"chainID":10,"latency":0,"gossipBlocks":true,"scores":{"gossip":{"total":0,"blocks":{"timeInMesh":0,"firstMessageDeliveries":0,"meshMessageDeliveries":0,"invalidMessageDeliveries":0},"IPColocationFactor":0,"behavioralPenalty":0},"reqResp":{"validResponses":0,"errorResponses":0,"rejectedPayloads":0}}},"16Uiu2HAmBsUrDQtp5om645VNu3Jr3J5ybtg4jwLPTudLAmGqdHZy":{"peerID":"16Uiu2HAmBsUrDQtp5om645VNu3Jr3J5ybtg4jwLPTudLAmGqdHZy","nodeID":"ed17fbb03fec6aac7724f12b8e9a68042e7bc821010a7c8d831e1830fb052a34","userAgent":"optimism","protocolVersion":"","ENR":"enr:-Jy4QDJlP-hjdigWz5pzHyud-7OXkbNF44bVwoLdIYO9TiPUHstetjhXHPX6dxZeCVrOermzIVJNnLq_cdamJA9xzH2GAY4Y3y1EgmlkgnY0gmlwhEFsLSKHb3BzdGFja4IKAIlzZWNwMjU2azGhAvRXgaGYOQXNOrTKF9U1qONnjGl-fIH8FBBJUHpDmWs4g3RjcIJ6h4N1ZHCCi28","addresses":["/ip4/65.108.45.34/tcp/31367/p2p/16Uiu2HAmBsUrDQtp5om645VNu3Jr3J5ybtg4jwLPTudLAmGqdHZy"],"protocols":["/floodsub/1.0.0","/ipfs/id/1.0.0","/ipfs/id/push/1.0.0","/ipfs/ping/1.0.0","/libp2p/autonat/1.0.0","/meshsub/1.0.0","/meshsub/1.1.0","/opstack/req/payload_by_number/10/0"],"connectedness":1,"direction":2,"protected":true,"chainID":10,"latency":0,"gossipBlocks":true,"scores":{"gossip":{"total":0.031111111111111114,"blocks":{"timeInMesh":15.40049799,"firstMessageDeliveries":0,"meshMessageDeliveries":0,"invalidMessageDeliveries":0},"IPColocationFactor":0,"behavioralPenalty":0},"reqResp":{"validResponses":0,"errorResponses":0,"rejectedPayloads":0}}},"16Uiu2HAmEYEPX2zeFhLWE92zkyPshFnuKWsfdvF4sxApF87pBpJ7":{"peerID":"16Uiu2HAmEYEPX2zeFhLWE92zkyPshFnuKWsfdvF4sxApF87pBpJ7","nodeID":"8f25c2d112b9f287f22f01ad9ad21b16feaccc5bb19262ed3a9d04a3b3593ed4","userAgent":"optimism","protocolVersion":"","ENR":"enr:-KS4QA6q63Et0tnyIjG2EDxyYtcOcD-OHSpZh67EG6nfaKUtdS6AALaaH8CgKonKM_E_GXOkIe53LyS9CUlFm9IfOgyGAY0xOhaogmlkgnY0gmlwhI1i2VaHb3BzdGFja4IKAIlzZWNwMjU2azGhAxv8LnYWJ_Bs3QpbYJJIKWiUUc2Ot9uuYsZU22p--K58g3RjcIIkBoN1ZHCCqoyEdWRwNoIkBg","addresses":["/ip4/141.98.217.86/tcp/9222/p2p/16Uiu2HAmEYEPX2zeFhLWE92zkyPshFnuKWsfdvF4sxApF87pBpJ7"],"protocols":["/floodsub/1.0.0","/ipfs/id/1.0.0","/ipfs/id/push/1.0.0","/ipfs/ping/1.0.0","/libp2p/autonat/1.0.0","/meshsub/1.0.0","/meshsub/1.1.0","/opstack/req/payload_by_number/10/0"],"connectedness":1,"direction":2,"protected":false,"chainID":10,"latency":1116561184,"gossipBlocks":true,"scores":{"gossip":{"total":0,"blocks":{"timeInMesh":0,"firstMessageDeliveries":0,"meshMessageDeliveries":0,"invalidMessageDeliveries":0},"IPColocationFactor":0,"behavioralPenalty":0},"reqResp":{"validResponses":0,"errorResponses":0,"rejectedPayloads":0}}},"16Uiu2HAmGQ7CXhVeGRNe8xcX5AH4jXyT6JFqA3H8gqZtL4Xc3zPQ":{"peerID":"16Uiu2HAmGQ7CXhVeGRNe8xcX5AH4jXyT6JFqA3H8gqZtL4Xc3zPQ","nodeID":"4d227487c4d27db28032d00ec53081c84f168ba1678048331adf90d9c2ceebf7","userAgent":"optimism","protocolVersion":"","ENR":"enr:-Jy4QBk-4-GZtmO349VtbBCSo9FU3OMqvhfWsLoSyJ5S4cv8MZ-npbHxJi_it8DjOCG2dsP4OZfczZDa08IErGcft9OGAY1x8zaugmlkgnY0gmlwhC1MH3iHb3BzdGFja4IKAIlzZWNwMjU2azGhAzeex5v2gXVo5QktqO3mcF1HXzSVLM3IVmvEXLauBY2Xg3RjcIIkBoN1ZHCCohw","addresses":["/ip4/45.76.31.120/tcp/9222/p2p/16Uiu2HAmGQ7CXhVeGRNe8xcX5AH4jXyT6JFqA3H8gqZtL4Xc3zPQ"],"protocols":["/libp2p/autonat/1.0.0","/meshsub/1.0.0","/meshsub/1.1.0","/opstack/req/payload_by_number/10/0","/floodsub/1.0.0","/ipfs/id/1.0.0","/ipfs/id/push/1.0.0","/ipfs/ping/1.0.0"],"connectedness":1,"direction":2,"protected":false,"chainID":10,"latency":247854740,"gossipBlocks":true,"scores":{"gossip":{"total":-5.04,"blocks":{"timeInMesh":0,"firstMessageDeliveries":0,"meshMessageDeliveries":0,"invalidMessageDeliveries":0},"IPColocationFactor":0,"behavioralPenalty":0},"reqResp":{"validResponses":0,"errorResponses":0,"rejectedPayloads":0}}},"16Uiu2HAmGdfRNRADxe1AHoGd2W5fXwibWDgJurLXN6zmVoRuqkkb":{"peerID":"16Uiu2HAmGdfRNRADxe1AHoGd2W5fXwibWDgJurLXN6zmVoRuqkkb","nodeID":"165c2e217870a8ee15b05af1f7aeed7ce69d4f45460f16a10efa8f5a2dc25c39","userAgent":"optimism","protocolVersion":"","ENR":"enr:-Jy4QEp1xjPZybIZRgtyO6RrLCQtvhIjCf-wjOOAmglUXxNndFJOJA6AEB4HF3mcy43rmz-vJVqiJA8AV9QZmERKu4eGAY31HUITgmlkgnY0gmlwhLI_Tl-Hb3BzdGFja4IKAIlzZWNwMjU2azGhAzsXw1iaPxeB-OadzpMjfBd28UwhVfo0iR4pmrUF6Cwsg3RjcIIkBoN1ZHCCJAY","addresses":["/ip4/178.63.78.95/tcp/9222/p2p/16Uiu2HAmGdfRNRADxe1AHoGd2W5fXwibWDgJurLXN6zmVoRuqkkb"],"protocols":["/meshsub/1.1.0","/opstack/req/payload_by_number/10/0","/floodsub/1.0.0","/ipfs/id/1.0.0","/ipfs/id/push/1.0.0","/ipfs/ping/1.0.0","/libp2p/autonat/1.0.0","/meshsub/1.0.0"],"connectedness":1,"direction":2,"protected":false,"chainID":10,"latency":0,"gossipBlocks":true,"scores":{"gossip":{"total":-5.04,"blocks":{"timeInMesh":0,"firstMessageDeliveries":0,"meshMessageDeliveries":0,"invalidMessageDeliveries":0},"IPColocationFactor":0,"behavioralPenalty":0},"reqResp":{"validResponses":0,"errorResponses":0,"rejectedPayloads":0}}},"16Uiu2HAmHNEAsu2zSM8cQbQAHT9xeeUENmJegtY82qk28feJ8nAT":{"peerID":"16Uiu2HAmHNEAsu2zSM8cQbQAHT9xeeUENmJegtY82qk28feJ8nAT","nodeID":"b06a7dc2223bdda13d42c779867c007ffb8d51740f564b87bf9cb98ecc0bb73c","userAgent":"optimism","protocolVersion":"","ENR":"enr:-Jy4QC_Yno8T2NxFfe1fVR-hoVmtEqVluYnM7i_n_7p-m9sLb_4p9_jMXIJ6cLr5ZCyzAxd3B6LDgELD56DHjLg23ZyGAY4fffBzgmlkgnY0gmlwhCPMCNyHb3BzdGFja4IKAIlzZWNwMjU2azGhA0X_MBDOyDipZSuORwbELEEXzK2BGvGVFRTEumSIg9TAg3RjcIIjK4N1ZHCCIys","addresses":["/ip4/35.204.8.220/tcp/9003/p2p/16Uiu2HAmHNEAsu2zSM8cQbQAHT9xeeUENmJegtY82qk28feJ8nAT"],"protocols":["/ipfs/ping/1.0.0","/libp2p/autonat/1.0.0","/meshsub/1.0.0","/meshsub/1.1.0","/opstack/req/payload_by_number/10/0","/floodsub/1.0.0","/ipfs/id/1.0.0","/ipfs/id/push/1.0.0"],"connectedness":1,"direction":2,"protected":false,"chainID":10,"latency":264584653,"gossipBlocks":true,"scores":{"gossip":{"total":0,"blocks":{"timeInMesh":0,"firstMessageDeliveries":0,"meshMessageDeliveries":0,"invalidMessageDeliveries":0},"IPColocationFactor":0,"behavioralPenalty":0},"reqResp":{"validResponses":0,"errorResponses":0,"rejectedPayloads":0}}},"16Uiu2HAmPfAXdPLYUexVZppwyZicSQdpsL7etcVfDoMrbLpZXrVW":{"peerID":"16Uiu2HAmPfAXdPLYUexVZppwyZicSQdpsL7etcVfDoMrbLpZXrVW","nodeID":"dae9f2827070c9c339a66706cdb0c2b1777a3fe9f0e138c4da3b599526131ac4","userAgent":"optimism","protocolVersion":"","ENR":"enr:-Jy4QKILpjb6wfvUZTrAGufE9YYTnpCFarObRDrzVT4tm_10RVLtAVtDUTJJT1YEpRXmBF6noygu1uM_yOu4El5bRQeGAY2mfyLPgmlkgnY0gmlwhJT7NsmHb3BzdGFja4IKAIlzZWNwMjU2azGhA6N8EJS5gAQ56D-n9Qt8egHlovBSGsRzXIIc9_JPuKIpg3RjcIIkBoN1ZHCC304","addresses":["/ip4/148.251.54.201/tcp/9222/p2p/16Uiu2HAmPfAXdPLYUexVZppwyZicSQdpsL7etcVfDoMrbLpZXrVW"],"protocols":["/meshsub/1.1.0","/opstack/req/payload_by_number/10/0","/floodsub/1.0.0","/ipfs/id/1.0.0","/ipfs/id/push/1.0.0","/ipfs/ping/1.0.0","/libp2p/autonat/1.0.0","/meshsub/1.0.0"],"connectedness":1,"direction":2,"protected":false,"chainID":10,"latency":0,"gossipBlocks":true,"scores":{"gossip":{"total":-5.04,"blocks":{"timeInMesh":0,"firstMessageDeliveries":0,"meshMessageDeliveries":0,"invalidMessageDeliveries":0},"IPColocationFactor":0,"behavioralPenalty":0},"reqResp":{"validResponses":0,"errorResponses":0,"rejectedPayloads":0}}},"16Uiu2HAmTrP55nReSSHC8seJ29DFg1vQKmFXhtdW4GsSkXBPgHSv":{"peerID":"16Uiu2HAmTrP55nReSSHC8seJ29DFg1vQKmFXhtdW4GsSkXBPgHSv","nodeID":"9fdbf15b858410597dc038353853c1ffdddd01ad0970075eb4e95e1df13eb5d7","userAgent":"optimism","protocolVersion":"","ENR":"enr:-Jy4QKeugr1KsGJi_Xf6OWdVl0xQnQC8sJV2Ccmf2ZlsXlzsfLmcNsqwr-hv9IPm382hg5OX_1kCSHaC4PQli6wrBM2GAYiWs8YIgmlkgnY0gmlwhIOZqU6Hb3BzdGFja4IKAIlzZWNwMjU2azGhA-HKYg5n7xVNhRiQgowp25JazsCt0vQctbrZDhEKPrvXg3RjcIIkBoN1ZHCCmj8","addresses":["/ip4/131.153.169.78/tcp/9222/p2p/16Uiu2HAmTrP55nReSSHC8seJ29DFg1vQKmFXhtdW4GsSkXBPgHSv"],"protocols":["/meshsub/1.0.0","/meshsub/1.1.0","/opstack/req/payload_by_number/10/0","/floodsub/1.0.0","/ipfs/id/1.0.0","/ipfs/id/push/1.0.0","/ipfs/ping/1.0.0","/libp2p/autonat/1.0.0"],"connectedness":1,"direction":2,"protected":false,"chainID":10,"latency":245993210,"gossipBlocks":true,"scores":{"gossip":{"total":-5.04,"blocks":{"timeInMesh":0,"firstMessageDeliveries":0,"meshMessageDeliveries":0,"invalidMessageDeliveries":0},"IPColocationFactor":0,"behavioralPenalty":0},"reqResp":{"validResponses":0,"errorResponses":0,"rejectedPayloads":0}}}},"bannedPeers":[],"bannedIPS":[],"bannedSubnets":[]}}

In light of this, I believe I should expose the P2P port and declare my public IP address. It seems that setting my NAT'd IP address in OP-NODE is required:

--p2p.advertise.ip value

Protocol Description

Version

op-geth: v1.101308.2
op-node: v1.7.1

op-geth args

    - --datadir
    - /data/op-geth
    - --rollup.sequencerhttp
    - https://mainnet-sequencer.optimism.io
    - --op-network
    - op-mainnet
    - --rollup.halt
    - major
    - --rollup.disabletxpoolgossip
    - --authrpc.addr
    - 0.0.0.0
    - --authrpc.port
    - "8551"
    - --authrpc.jwtsecret
    - /secret/jwtsecret
    - --authrpc.vhosts=*
    - --http
    - --http.api
    - web3,eth,net,debug
    - --http.addr
    - 0.0.0.0
    - --http.port
    - "8545"
    - --http.vhosts=*
    - --http.corsdomain=*
    - --ws
    - --ws.addr
    - 0.0.0.0
    - --ws.port
    - "8546"
    - --ws.origins=*
    - --metrics
    - --metrics.expensive
    - --pprof
    - --pprof.addr
    - 0.0.0.0
    - --syncmode
    - full
    - --snapshot=false
    - --maxpeers
    - "0"
    - --nodiscover
    - --history.transactions
    - "0"

op-node args

    - op-node
    - --l1
    - http://ethereum-mainnet-api:8599
    - --l2
    - http://localhost:8551
    - --l1.beacon=http://ethereum2-mainnet-api:8080
    - --l2.jwt-secret
    - /secret/jwtsecret
    - --log.level
    - info
    - --network
    - op-mainnet
    - --rollup.load-protocol-versions
    - "true"
    - --rollup.halt
    - major
    - --rpc.addr
    - 0.0.0.0
    - --rpc.port
    - "9545"
    - --metrics.enabled
    - --p2p.listen.tcp
    - "9222"

Node Logs

No response

Additional Information

No response

Feedback

No response

[imtipi]

you didn't run in snap sync(which means p2p)
please check this https://docs.optimism.io/builders/node-operators/management/snap-sync

[J1a-wei]

Hi @imtipi
This is a log I opened. It has been running for two days and seems to be continuously generating snapshots.

WARN [03-16|04:43:43.590] Ignoring already known beacon payload number=117,479,216 hash=ae7974..881fe0 age=1h56m54s
WARN [03-16|04:43:43.590] Ignoring already known beacon payload number=117,479,216 hash=ae7974..881fe0 age=1h56m54s
WARN [03-16|04:43:47.100] Ignoring already known beacon payload number=117,479,216 hash=ae7974..881fe0 age=1h56m58s
INFO [03-16|04:43:47.104] Chain head was updated number=117,479,216 hash=ae7974..881fe0 root=816d9c..5a7f6f elapsed=2.762204ms age=1h56m58s
INFO [03-16|04:43:47.121] Aborting state snapshot generation root=1e8b2c..504649 in=08aa01..80cc64 at=f2e622..738d69 accounts=4,131,617 slots=8,361,573 storage=913.95MiB dangling=0 elapsed=44h20m36.227s eta=1265h52m57.789s
INFO [03-16|04:43:47.121] Resuming state snapshot generation root=47dac7..f756e5 in=08aa01..80cc64 at=f2e622..738d69 accounts=4,131,617 slots=8,361,573 storage=913.95MiB dangling=0 elapsed=44h20m36.228s eta=1265h52m57.819s
INFO [03-16|04:43:47.123] Imported new potential chain segment number=117,479,217 hash=1901cc..396eee blocks=1 txs=1 mgas=0.052 elapsed=15.048ms mgasps=3.471 age=1h56m56s snapdiffs=97.12KiB triedirty=184.70MiB
INFO [03-16|04:43:47.124] Chain head was updated number=117,479,217 hash=1901cc..396eee root=2b7afa..e84431 elapsed="135.187µs" age=1h56m56s
INFO [03-16|04:43:47.715] Aborting state snapshot generation root=47dac7..f756e5 in=08aa01..80cc64 at=f2e729..d21d3d accounts=4,131,617 slots=8,361,603 storage=913.95MiB dangling=0 elapsed=44h20m36.822s eta=1265h53m14.776s
INFO [03-16|04:43:47.716] Resuming state snapshot generation root=60dfb4..8d5a48 in=08aa01..80cc64 at=f2e729..d21d3d accounts=4,131,617 slots=8,361,603 storage=913.95MiB dangling=0 elapsed=44h20m36.822s eta=1265h53m14.776s
INFO [03-16|04:43:47.720] Imported new potential chain segment number=117,479,218 hash=d305a5..15d43e blocks=1 txs=29 mgas=6.487 elapsed=592.995ms mgasps=10.940 age=1h56m54s snapdiffs=100.06KiB triedirty=184.92MiB
INFO [03-16|04:43:47.722] Chain head was updated number=117,479,218 hash=d305a5..15d43e root=a14eed..6075de elapsed="721.062µs" age=1h56m54s

I am curious if syncmode full and snap will ultimately ensure the same data volume. In other words, both modes will lead to having "full nodes", but the synchronization method varies; syncmode allows syncing from other nodes. Right?

Additionally, my node has a base, which also uses the op stack technology stack. However, the base's network seems to perform better; even without using syncmode=snap, it can catch up to the latest blocks.

[imtipi]

yes,snap sync allow you sync from other nodes

[Chomtana]

If you use an advanced firewall/routing system you should set --p2p.advertise.ip on op-node as you have stated.

I am not sure about this, but I think you may need to recheck your NAT translation/firewall rules.

Also, you may need to set --p2p.listen.udp=9222 on op-node.

If it's working correctly, you should see Received signed execution payload from p2p in the op-node log.

[1195597989]

Maybe you should check your network and firewall