[Node Operator Question] After upgrading the node, it has been continuously looking for peers

[dpguoming]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

node stopped sync with Looking for peers after upgrade 1.7.4

Protocol Description

op-node to v1.7.3
op-geth to v1.101311.0.

Node Logs

INFO [04-16|10:00:38.834] Looking for peers                        peercount=0 tried=160 static=0
INFO [04-16|10:00:49.728] Looking for peers                        peercount=0 tried=46  static=0
INFO [04-16|10:01:00.410] Looking for peers                        peercount=0 tried=105 static=0
INFO [04-16|10:01:10.442] Looking for peers                        peercount=0 tried=84  static=0
INFO [04-16|10:01:20.484] Looking for peers                        peercount=0 tried=103 static=0
INFO [04-16|10:01:26.467] New local node record                    seq=1,694,117,146,213 id=46e19b80538f7408 ip=44.196.208.70 udp=43632 tcp=39393
INFO [04-16|10:01:30.489] Looking for peers                        peercount=0 tried=102 static=0

Additional Information

No response

Feedback

No response

[imtipi]

can you post your start command here and make sure port 9003 opened

[dpguoming]

I use docker run the node, port 9003 is opened.

op-node --l1=http://127.0.0.1:8555 --l2=http://op-geth:8551 --rpc.addr=127.0.0.1 --rpc.port=9545 --l2.jwt-secret=/shared/jwt.txt --l1.trustrpc --l1.rpckind=basic --l1.beacon=https://127.0.0.1:8666 --metrics.enabled --metrics.addr=0.0.0.0 --metrics.port=7300 --syncmode=execution-layer --network=op-mainnet --rollup.load-protocol-versions=true --rollup.halt=major

geth --datadir=/geth --http --http.corsdomain=* --http.vhosts=* --http.addr=0.0.0.0 --http.port=8545 --http.api=web3,debug,eth,txpool,net,engine --ws --ws.addr=0.0.0.0 --ws.port=8546 --ws.origins=* --ws.api=debug,eth,txpool,net,engine,web3 --metrics --metrics.influxdb --metrics.influxdb.endpoint=http://influxdb:8086 --metrics.influxdb.database=opgeth --syncmode=snap --gcmode=full --authrpc.vhosts=* --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.jwtsecret=/shared/jwt.txt --rollup.sequencerhttp=https://mainnet-sequencer.optimism.io --rollup.disabletxpoolgossip=true --port=39393 --discovery.port=39393 --rollup.historicalrpc=https://mainnet.optimism.io --op-network=op-mainnet

[imtipi]

--port=39393 --discovery.port=39393
do you also opened port 39393?

[dpguoming]

all port is opened in docker router

23de1d4b746e   us-docker.pkg.dev/oplabs-tools-artifacts/images/op-geth:v1.101311.0   "/scripts/start-op-g…"   8 hours ago     Up 8 hours                      30303/tcp, 30303/udp, 0.0.0.0:39393->39393/tcp, 0.0.0.0:39393->39393/udp, :::39393->39393/tcp, :::39393->39393/udp, 0.0.0.0:9991->8545/tcp, :::9991->8545/tcp, 0.0.0.0:9992->8546/tcp, :::9992->8546/tcp                        simple-optimism-node-op-geth-1
b049a24c9eff   us-docker.pkg.dev/oplabs-tools-artifacts/images/op-node:v1.7.3        "/scripts/start-op-n…"   8 hours ago     Up 8 hours                      0.0.0.0:9003->9003/tcp, 0.0.0.0:9003->9003/udp, :::9003->9003/tcp, :::9003->9003/udp                                                                                                                                            simple-optimism-node-op-node-1

Is it possible caused by opnode restarting "Walking back L1Block by hash"? And some other err logs

t=2024-04-17T00:58:58+0000 lvl=warn msg="failed to notify engine driver of L1 head change" err="context deadline exceeded"
t=2024-04-17T01:00:32+0000 lvl=warn msg="failed to notify engine driver of new L2 payload" err="context deadline exceeded" 

[imtipi]

Is it possible caused by opnode restarting "Walking back L1Block by hash"? And some other err logs

can be,please also provide both opnode and opgeth log

[dpguoming]

• node just started

t=2024-04-17T21:18:46+0000 lvl=info msg="Walking back L1Block by hash" curr=0xe2e106edd41eb6887e20b5501ab3db30b909db5c2a903fdafba804867a0e88ac:19649375 next=0x766ee8b1430914dab6561ff1860be619f7ffe8423a87b4ec321ee21af6280ef8:19649374 l2block=0xe70d71f2c761664f521fa5535cf1cc4c6aea5145d669e90892f251c3a49ea359:118722852

• node started after a few hour

t=2024-04-17T20:59:53+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmRedmzVCu3LwD71Gg39Wv2BW2uGFzfvr82JkG2w73k2MN
t=2024-04-17T20:59:53+0000 lvl=info msg="Received signed execution payload from p2p" id=0x0d504d126ae0125a2d6702d7cefc111f049c7bde3213665426c1c462b7f2d45a:118894408 peer=16Uiu2HAmHNEAsu2zSM8cQbQAHT9xeeUENmJegtY82qk28feJ8nAT
t=2024-04-17T20:59:53+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x0d504d126ae0125a2d6702d7cefc111f049c7bde3213665426c1c462b7f2d45a:118894408
t=2024-04-17T20:59:53+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm4E53jPzDKAtDrC2c25UkZLFEoC5wiRZgdXrwZWgf3GtA
t=2024-04-17T20:59:54+0000 lvl=warn msg="Derivation process temporary error" attempts=1619 err="engine stage failed: temp: failed to fetch blobs: failed to get blob sidecars for L1BlockRef 0xda4d94cc0f3ebd5447c858790dcefdca4d6a39623eee4609e3f287f2501dd6dc:19429617: failed to fetch blob sidecars for slot 8629261 block 0xda4d94cc0f3ebd5447c858790dcefdca4d6a39623eee4609e3f287f2501dd6dc:19429617: failed request with status 500: {\"code\":500,\"message\":\"block not found\"}"

t=2024-04-18T00:51:30+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x311f2e5e7fea3941baceda77288700b06dae44e978b3a33b45172e6b4880c92e:118901356
t=2024-04-18T00:51:30+0000 lvl=info msg="Advancing bq origin" origin=0xb5c37c680b80606ac398885abd2a459756f1e7642ebc1f8ce4bf20fcc3ee6b6b:19436480 originBehind=false
t=2024-04-18T00:51:30+0000 lvl=info msg="Advancing bq origin" origin=0x8738227455bae3f26cc940aed6c00156c154752632ae06ea262dbe780890a957:19436481 originBehind=false
t=2024-04-18T00:51:30+0000 lvl=info msg="disconnected from peer" peer=16Uiu2HAmBro3LNxsREmabTYtMXwJqnNTqtxMSB96Q9DKWjmpaKJh addr=/ip4/65.109.81.34/tcp/9222
t=2024-04-18T00:51:31+0000 lvl=info msg="Advancing bq origin" origin=0x9325badb0c4fc4e9951727865bf577ec16ec76d058a9daa0f842b1202f8fbf81:19436482 originBehind=false
t=2024-04-18T00:51:31+0000 lvl=info msg="Advancing bq origin" origin=0x75eaf8711afdd7ff042ef151c7cac5a146f85fcc0dd4d13ed91316bab5a9877f:19436483 originBehind=false
t=2024-04-18T00:51:31+0000 lvl=info msg="Advancing bq origin" origin=0xc06243ce73dd2a70214ef45b0bb9d937bf98a3163df44ba8c453908344c316ed:19436484 originBehind=false
t=2024-04-18T00:51:31+0000 lvl=info msg="Advancing bq origin" origin=0x9140bdb325f296f834fca1d1558971d70c0da48f3d64862c056d00091b7e8be2:19436485 originBehind=false
t=2024-04-18T00:51:32+0000 lvl=info msg="Advancing bq origin" origin=0x81e72c41cd91842a8fe728f4e999e3ab27b1fcc6975ac1a9d52053cedea369e9:19436486 originBehind=false
t=2024-04-18T00:51:32+0000 lvl=info msg="Advancing bq origin" origin=0xbd59b201210f8e223b678229f2a6f432bd6fa5acfd1662003aeb593de4cf4f74:19436487 originBehind=false
t=2024-04-18T00:51:32+0000 lvl=info msg="Received signed execution payload from p2p" id=0xb09b85cfe39fd4a6ef1b714b32f36e1bd3341b1cfae98dd3310c51490b7330dd:118901357 peer=16Uiu2HAkwqPxTqTVWeL3KFKHtbhYfAmuyT5LELZudrAhf1i3qNg2
t=2024-04-18T00:51:32+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xb09b85cfe39fd4a6ef1b714b32f36e1bd3341b1cfae98dd3310c51490b7330dd:118901357
t=2024-04-18T00:51:32+0000 lvl=info msg="Advancing bq origin" origin=0xd3e703c9820913e7fd64f42ce9113d4bfe21ed74ff44dce89f7456f568220ee9:19436488 originBehind=false
t=2024-04-18T00:51:33+0000 lvl=info msg="Advancing bq origin" origin=0xc6cd5261fb32695b2677b05c72c0c0e31e8a046971f05ae7f48f147e93792cb2:19436489 originBehind=false
t=2024-04-18T00:51:33+0000 lvl=warn msg="tx in inbox with unauthorized submitter" addr=0x6088B06c5a187058434655B71057a9ee93E13d0d hash=0xe1907406559db89f8432d6118c117220d0cd42372a231aeb60a6e06d89ad5ddc err=<nil>
t=2024-04-18T00:51:33+0000 lvl=info msg="Advancing bq origin" origin=0x896be0df305dbc6424bcf36082d2c5098965c5b2214f9746bc1681a604778247:19436490 originBehind=false
t=2024-04-18T00:51:33+0000 lvl=info msg="Advancing bq origin" origin=0xcf9612cd487b5c57ce374458c39a6f390aa483ae9a693fb4888c04786169beff:19436491 originBehind=false
t=2024-04-18T00:51:33+0000 lvl=info msg="created new channel" origin=0xcf9612cd487b5c57ce374458c39a6f390aa483ae9a693fb4888c04786169beff:19436491 channel=08b04f5d712414b91d71eb8adbadaec9 length=130020 frame_number=0 is_last=false
t=2024-04-18T00:51:33+0000 lvl=info msg="Reading channel" channel=08b04f5d712414b91d71eb8adbadaec9 frames=6
t=2024-04-18T00:51:33+0000 lvl=info msg="Found next batch" batch_type=SpanBatch batch_timestamp=1710458327 parent_check=0x4a97c2997e289f4c1817beb3e6ed5a3d1e65e743 origin_check=0x9325badb0c4fc4e9951727865bf577ec16ec76d0 start_epoch_number=19436433 end_epoch_number=19436482 block_count=308
t=2024-04-18T00:51:33+0000 lvl=info msg="generated attributes in payload queue" txs=11 timestamp=1710458327
t=2024-04-18T00:51:34+0000 lvl=info msg="Sync progress" reason="reconciled with L1" l2_finalized=0x513e8eb63151ddcb202453b0dd214f96ea70758076c679a84d24206c85a44fa0:117429482 l2_safe=0x4a97c2997e289f4c1817beb3e6ed5a3d1e65e7434c8ba30c58fdd802d9f87057:117429774 l2_pending_safe=0x2688dfa53c4c5ed51622af230aa26ca7186ae4b62843e644aa02605f4d74a3e6:117429775 l2_unsafe=0xb34efeb9758622d258919097ad0986edc89d3c5b15749d092969a12b16c04af4:118815551 l2_backup_unsafe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_time=1713229879 l1_derived=0xcf9612cd487b5c57ce374458c39a6f390aa483ae9a693fb4888c04786169beff:19436491

• geth

INFO [04-17|21:17:27.287] Looking for peers                        peercount=0 tried=79  static=0

[dpguoming]

I replace OP_NODE__RPC_ENDPOINT and OP_NODE__L1_BEACON to quicknode, now is starting sync

[dpguoming]

After review the op-node logs, it firstly walking back to 115649693, then start sync. Walking back will take long time, and sync to latest op-node block may last for several days when slow l1 node access.