Reth BSC Testnet nodes stops syncing very often #90
Description
Describe the bug
Very often we have one of our Reth BSC Testnet nodes stopped syncing.
I've pasted all the logs. Nodes stopped ~23:23UTC.
Resources look fine during that time, the only thing we see in metrics is that numbers of peers slightly dropped for that one (even before it stopped), and continue to drop after.
We have this setup:
- --trusted-peers=enode://1cc5bc6bab2fa9f6e6f68e9089ffe2664662b9dd6c78b635b327e1510260eb64117130cb3c08898c7f438a7efd849eb83134f7097ef31ba06967de65f8bffd8f@213.35.126.94:30303,enode://caba45f6d9c479c821ca979f79c34aeb5e86e1a6214971ee381d54323e6f6902e51c8ae61ab33c5f8377db80a1bd529c63b623290f794ecef69a8240b63e40ab@142.44.212.106:30311,enode://502a9bfe3860835d6ca6de46d44f1c10dcb6be3ae53f5ba6e14d8e9db337de84e2cfb7d08485db7f85ff805d23a849d3d837c945f6f7a3f3775d1604956bf0ee@64.130.45.158:30303,enode://eb789212dc7d4fb67480c25dc7ffb9b709c5750efb12ff41f4c8347503179f7d1ffae990b10c4a4f2f54dfeb7a97e67ba84273b5cec8521ba9f8b68e3abc1bca@23.88.70.181:12759,enode://706d99e35841db3dea3cbc3025c8864050394fb741a636e3175e582e4f5ae307bbb8897cc4966cd046851e09cb1727217d8ada13876b8b46b07dc3ce65095608@74.63.203.118:30303,enode://213958a282866806b55e34b4b7001c9e6732fc007452d13628914e3cc484ff2ae7af2df5baec4de09cba5c5751b770304ed4e62a629f4e8371acdc527299d4c3@217.22.153.165:31615,enode://e5c4320eaa3357286cdde303df8b5b84f81013d86a72f91ecb2efc59b48a376bf16904d0a4e8ca44981c8d201bef439e1fb91c551d24aa39b65d930f03fc1823@52.51.80.128:30311,enode://8bfa647afc442a534080104ae5fe94a07efc2577f5ac617f7d7fefc67128b2797633ca52810ec31920588e96b31fafd88eda3c66e9ddc421f9ae796e7810477a@61.111.3.77:20131,enode://b4db5861cb1351a3c187f307ff7cd8994e7a7979fb008c8123eaed51cebdd0125424c7b9386ad9cbde920a59c378829c4aff9da213c730c1efe9932a4a7f4713@162.19.43.146:30051,enode://75601809401e4dedf6477fa9b74170d932b76aba0d1de1c19b27ff0a424ede294b5fc235af64f41dd4003a43793f63f321082b4de6d6a0588b5c84215f909af9@3.209.122.123:30311,enode://ac44219f421dc2ff0b164822383663306d5affdac628dc7308f03b7b9fc50a685107b7df89d7fec6ef3de8ba3c2d488a185fa413b8b1126c6337852ed4938380@51.178.74.129:30311,enode://8e999f9e898b1cf8494ed4a5dc3627384fc159bdea16a9a3e8f2fd517d44f178e346f86a67009af7e3108da5c87b7f8a08d787e819db9f511931e8adb05d2038@91.134.60.248:6052,enode://ab7b5817cd36fdf4acf639c069cdebf93f7e24ab22776c1005ce7d1309407dfde5ad7fd66e4b721a8d15fade240917378f67ecd618789cd784da149c2e550560@109.173.126.247:30312,enode://3bdd3b0e014ce1c5dc17f069459448f4526c357ab655470614a5d682a92e8f6e86d6fafacbf4e52bdb2c808b0d91fa0b95621f07a4667fc4c3d01a1b7b736060@162.19.232.12:30303,enode://cbb7c21f55f2fe081e726abb8612f908f63e6b2b663ba6e981ec9d0c76302198a31b07c599aaf532f32593ddb518ab56677cec7506d09d21145963cfb522506c@47.76.150.188:30309,enode://c543c28760aacd710001803354fefd6e916372629acfd2c3cbc0aa5670b65ae4adc2fed2f81fbe1c8d4122cc0f170e074f5e1109e26416e213eecad3ae2de942@135.181.130.229:30303,enode://f677529b108e4c24b1f6a69c2101fed6604d5b86c40f2bf7dad0c286f6dd8753f3cc6e1d1e3e61a282370b15e2c85c2091da565f4bb36b92e9ead0af34c1dcf3@101.47.139.54:30304,enode://6a9ab069a48937c1d4a83e1a0000b0900a18d1ebc04e9bf30031b13c05d78685c806d2aa6c3ce8098bb0a82eaa7d18056e783877c058487c68bd095f8a34becb@150.136.66.106:30303,enode://3e185dfce1cecb4c6e58257cc2f26666d23037f0f49c8a8bd4e4ed3260307179346f2ce641ef4adf500bce7280c8b3eb3c09605979220199968d934c71e9a37f@58.151.99.238:30311,enode://71af80890b204572fa3ab7c0f0828c7aa55e824e7e66a6bba14e0499cc93b2724ed9e66906f64a7f43b8ff6a3fe6dfd52ad6800975742df18360e71f1b32ea18@87.246.108.80:31616,enode://665cf77ca26a8421cfe61a52ac312958308d4912e78ce8e0f61d6902e4494d4cc38f9b0dd1b23a427a7a5734e27e5d9729231426b06bb9c73b56a142f83f6b68@52.72.123.113:30311,enode://27e979e9da2368477af1e8214397ff3517fdbf3b0921575584e2d6fad2eeafdaab6f162dcafc8db47806100abeff68d435277402d0f2dde4a92850bc6a1307b0@47.130.184.61:30303
Steps to reproduce
Start node and wait until it stops.
Node logs
2026-01-29T23:22:54.558445Z INFO Received headers total=117 from_block=87337373 to_block=87337257
2026-01-29T23:22:54.558611Z INFO Writing headers total=117
2026-01-29T23:22:54.558633Z INFO Executing stage pipeline_stages=2/13 stage=Headers checkpoint=87337256 target=0
2026-01-29T23:22:54.559076Z INFO Writing headers progress=9.40%
2026-01-29T23:22:54.559190Z INFO Writing headers progress=18.80%
2026-01-29T23:22:54.559286Z INFO Writing headers progress=28.21%
2026-01-29T23:22:54.559464Z INFO Writing headers progress=37.61%
2026-01-29T23:22:54.559551Z INFO Writing headers progress=47.01%
2026-01-29T23:22:54.559644Z INFO Writing headers progress=56.41%
2026-01-29T23:22:54.559732Z INFO Writing headers progress=65.81%
2026-01-29T23:22:54.559822Z INFO Writing headers progress=75.21%
2026-01-29T23:22:54.559903Z INFO Writing headers progress=84.62%
2026-01-29T23:22:54.559973Z INFO Writing headers progress=94.02%
2026-01-29T23:22:54.560010Z INFO Writing headers hash index total=117
2026-01-29T23:22:54.560920Z INFO Writing headers hash index progress=9.40%
2026-01-29T23:22:54.561455Z INFO Writing headers hash index progress=18.80%
2026-01-29T23:22:54.562230Z INFO Writing headers hash index progress=28.21%
2026-01-29T23:22:54.562955Z INFO Writing headers hash index progress=37.61%
2026-01-29T23:22:54.563542Z INFO Writing headers hash index progress=47.01%
2026-01-29T23:22:54.564089Z INFO Writing headers hash index progress=56.41%
2026-01-29T23:22:54.564874Z INFO Writing headers hash index progress=65.81%
2026-01-29T23:22:54.566090Z INFO Writing headers hash index progress=75.21%
2026-01-29T23:22:54.567167Z INFO Writing headers hash index progress=84.62%
2026-01-29T23:22:54.567915Z INFO Writing headers hash index progress=94.02%
2026-01-29T23:22:54.574232Z INFO Downloading bodies count=117 range=87337257..=87337373
2026-01-29T23:22:54.574230Z INFO Finished stage pipeline_stages=2/13 stage=Headers checkpoint=87337373 target=0 stage_progress=100.00%
2026-01-29T23:22:54.574260Z INFO Preparing stage pipeline_stages=3/13 stage=Bodies checkpoint=87337256 target=87337373
2026-01-29T23:22:54.863163Z INFO Executing stage pipeline_stages=3/13 stage=Bodies checkpoint=87337256 target=87337373
2026-01-29T23:22:54.871539Z INFO Recovering senders tx_range=627520301..627521471
2026-01-29T23:22:54.871533Z INFO Finished stage pipeline_stages=3/13 stage=Bodies checkpoint=87337373 target=87337373 stage_progress=100.00%
2026-01-29T23:22:54.871567Z INFO Preparing stage pipeline_stages=4/13 stage=SenderRecovery checkpoint=87337256 target=87337373
2026-01-29T23:22:54.871573Z INFO Executing stage pipeline_stages=4/13 stage=SenderRecovery checkpoint=87337256 target=87337373
2026-01-29T23:22:54.884256Z INFO Finished stage pipeline_stages=4/13 stage=SenderRecovery checkpoint=87337373 target=87337373 stage_progress=100.00%
2026-01-29T23:22:54.884279Z INFO Preparing stage pipeline_stages=5/13 stage=Execution checkpoint=87337256 target=87337373
2026-01-29T23:22:54.884306Z INFO Executing stage pipeline_stages=5/13 stage=Execution checkpoint=87337256 target=87337373
2026-01-29T23:22:55.353349Z INFO Finished stage pipeline_stages=5/13 stage=Execution checkpoint=87337373 target=87337373 stage_progress=100.00%
2026-01-29T23:22:55.353378Z INFO Preparing stage pipeline_stages=6/13 stage=MerkleUnwind checkpoint=87337256 target=87337373
2026-01-29T23:22:55.353392Z INFO Stage is always skipped
2026-01-29T23:22:55.353400Z INFO Executing stage pipeline_stages=6/13 stage=MerkleUnwind checkpoint=87337256 target=87337373
2026-01-29T23:22:55.354110Z INFO Finished stage pipeline_stages=6/13 stage=MerkleUnwind checkpoint=87337373 target=87337373
2026-01-29T23:22:55.354118Z INFO Preparing stage pipeline_stages=7/13 stage=AccountHashing checkpoint=87337256 target=87337373
2026-01-29T23:22:55.354151Z INFO Executing stage pipeline_stages=7/13 stage=AccountHashing checkpoint=87337256 target=87337373
2026-01-29T23:22:55.375228Z INFO Finished stage pipeline_stages=7/13 stage=AccountHashing checkpoint=87337373 target=87337373 stage_progress=100.00%
2026-01-29T23:22:55.375243Z INFO Preparing stage pipeline_stages=8/13 stage=StorageHashing checkpoint=87337256 target=87337373
2026-01-29T23:22:55.375274Z INFO Executing stage pipeline_stages=8/13 stage=StorageHashing checkpoint=87337256 target=87337373
2026-01-29T23:22:55.469633Z INFO Finished stage pipeline_stages=8/13 stage=StorageHashing checkpoint=87337373 target=87337373 stage_progress=100.00%
2026-01-29T23:22:55.469664Z INFO Preparing stage pipeline_stages=9/13 stage=MerkleExecute checkpoint=87337256 target=87337373
2026-01-29T23:22:55.469677Z INFO Executing stage pipeline_stages=9/13 stage=MerkleExecute checkpoint=87337256 target=87337373
2026-01-29T23:22:55.830542Z INFO Finished stage pipeline_stages=9/13 stage=MerkleExecute checkpoint=87337373 target=87337373 stage_progress=100.00%
2026-01-29T23:22:55.830574Z INFO Updating transaction lookup tx_range=87337256..=87337373
2026-01-29T23:22:55.830578Z INFO Preparing stage pipeline_stages=10/13 stage=TransactionLookup checkpoint=87337256 target=87337373
2026-01-29T23:22:55.830596Z INFO Executing stage pipeline_stages=10/13 stage=TransactionLookup checkpoint=87337256 target=87337373
2026-01-29T23:22:55.830599Z INFO Calculating transaction hashes tx_range=627520301..627521471
2026-01-29T23:22:55.860043Z INFO Inserting hashes append_only=false progress=10.00%
2026-01-29T23:22:55.886730Z INFO Inserting hashes append_only=false progress=20.00%
2026-01-29T23:22:55.912962Z INFO Inserting hashes append_only=false progress=30.00%
2026-01-29T23:22:55.939873Z INFO Inserting hashes append_only=false progress=40.00%
2026-01-29T23:22:55.967076Z INFO Inserting hashes append_only=false progress=50.00%
2026-01-29T23:22:55.994224Z INFO Inserting hashes append_only=false progress=60.00%
2026-01-29T23:22:56.021074Z INFO Inserting hashes append_only=false progress=70.00%
2026-01-29T23:22:56.048631Z INFO Inserting hashes append_only=false progress=80.00%
2026-01-29T23:22:56.075490Z INFO Inserting hashes append_only=false progress=90.00%
2026-01-29T23:22:56.139060Z INFO Received block from consensus engine number=87337413 hash=0x328572d4ace91c7ec6810bd2057c7821a1d1d1128fc94865815ffd5becc3d80b
2026-01-29T23:22:56.151011Z INFO Finished stage pipeline_stages=10/13 stage=TransactionLookup checkpoint=87337373 target=87337373 stage_progress=100.00%
2026-01-29T23:22:56.151025Z INFO Collecting indices first_sync=false
2026-01-29T23:22:56.151033Z INFO Preparing stage pipeline_stages=11/13 stage=IndexStorageHistory checkpoint=87337256 target=87337373
2026-01-29T23:22:56.151039Z INFO Executing stage pipeline_stages=11/13 stage=IndexStorageHistory checkpoint=87337256 target=87337373
2026-01-29T23:22:56.152305Z INFO Loading indices into database
2026-01-29T23:22:56.175400Z INFO Writing indices progress=9.98%
2026-01-29T23:22:56.181774Z INFO Writing indices progress=19.96%
2026-01-29T23:22:56.201982Z INFO Writing indices progress=29.93%
2026-01-29T23:22:56.207755Z INFO Writing indices progress=39.91%
2026-01-29T23:22:56.213050Z INFO Writing indices progress=49.89%
2026-01-29T23:22:56.218047Z INFO Writing indices progress=59.87%
2026-01-29T23:22:56.223834Z INFO Writing indices progress=69.85%
2026-01-29T23:22:56.230222Z INFO Writing indices progress=79.83%
2026-01-29T23:22:56.236295Z INFO Writing indices progress=89.80%
2026-01-29T23:22:56.252281Z INFO Writing indices progress=99.78%
2026-01-29T23:22:56.276860Z INFO Collecting indices first_sync=false
2026-01-29T23:22:56.276859Z INFO Finished stage pipeline_stages=11/13 stage=IndexStorageHistory checkpoint=87337373 target=87337373
2026-01-29T23:22:56.276883Z INFO Preparing stage pipeline_stages=12/13 stage=IndexAccountHistory checkpoint=87337256 target=87337373
2026-01-29T23:22:56.276898Z INFO Executing stage pipeline_stages=12/13 stage=IndexAccountHistory checkpoint=87337256 target=87337373
2026-01-29T23:22:56.277251Z INFO Loading indices into database
2026-01-29T23:22:56.280423Z INFO Writing indices progress=9.78%
2026-01-29T23:22:56.282653Z INFO Writing indices progress=19.56%
2026-01-29T23:22:56.284094Z INFO Writing indices progress=29.33%
2026-01-29T23:22:56.285997Z INFO Writing indices progress=39.11%
2026-01-29T23:22:56.289823Z INFO Writing indices progress=48.89%
2026-01-29T23:22:56.292532Z INFO Writing indices progress=58.67%
2026-01-29T23:22:56.294398Z INFO Writing indices progress=68.44%
2026-01-29T23:22:56.295643Z INFO Writing indices progress=78.22%
2026-01-29T23:22:56.297398Z INFO Writing indices progress=88.00%
2026-01-29T23:22:56.300252Z INFO Writing indices progress=97.78%
2026-01-29T23:22:56.308954Z INFO Finished stage pipeline_stages=12/13 stage=IndexAccountHistory checkpoint=87337373 target=87337373
2026-01-29T23:22:56.308983Z INFO Preparing stage pipeline_stages=13/13 stage=Finish checkpoint=87337256 target=87337373
2026-01-29T23:22:56.308989Z INFO Executing stage pipeline_stages=13/13 stage=Finish checkpoint=87337256 target=87337373
2026-01-29T23:22:56.309363Z INFO Finished stage pipeline_stages=13/13 stage=Finish checkpoint=87337373 target=87337373
2026-01-29T23:22:56.928629Z INFO Received block from consensus engine number=87337414 hash=0xc85c1a2a5fe275105d4d9f475266759322a94e41730c0da308132e4725050ffe
2026-01-29T23:22:57.170665Z INFO Received block from consensus engine number=87337415 hash=0x519f62132ceddf5162cf5f1d1383c96f8e5e73b6f1b247fcde8439a913b9176d
2026-01-29T23:22:58.356808Z INFO Preparing stage pipeline_stages=1/13 stage=Era checkpoint=0 target=None
2026-01-29T23:22:58.356833Z INFO Executing stage pipeline_stages=1/13 stage=Era checkpoint=0 target=None
2026-01-29T23:22:58.356839Z INFO Finished stage pipeline_stages=1/13 stage=Era checkpoint=0 target=None
2026-01-29T23:22:58.356844Z INFO Preparing stage pipeline_stages=2/13 stage=Headers checkpoint=87337373 target=0
2026-01-29T23:22:59.402811Z INFO Received block from consensus engine number=87337359 hash=0xec7ce3c30cf6c8d144c80b9f39c6e00c4ef6564c09c06f18a41a0b366ca9298f
2026-01-29T23:22:59.404840Z INFO Received block from consensus engine number=87337401 hash=0x404de4d24cb1e3d392d86764878728f692df4dcce595a986002ef1c1003317b0
2026-01-29T23:22:59.405415Z INFO Received block from consensus engine number=87337403 hash=0xf5a9050efba433f0fa1bcea4681827912717c7df5c6eb7d60a4aa76831d73dc1
2026-01-29T23:22:59.405983Z INFO Received block from consensus engine number=87337412 hash=0x2bf711c528de799530c18e16f2b828e6b16ff262af35b324fa778f2bdbb44fd7
2026-01-29T23:23:00.193983Z INFO Received block from consensus engine number=87337406 hash=0x71a2fd36b2d42fad0fcb30db1dda224d8d42be69327851a2d8bd62190e4631f3
2026-01-29T23:23:02.111837Z INFO Received block from consensus engine number=87337417 hash=0xc108ab903a49fa5d9396c8f7aea8e13eff3cc2cb3e2662ef0c5c60e59d5c6c17
2026-01-29T23:23:02.115664Z INFO Received block from consensus engine number=87337418 hash=0xb5e2612921385fda90404271394e130220fecf2d03e195b0d1ffc94d9854f280
2026-01-29T23:23:03.315827Z WARN Global signer not initialized for mining mode
2026-01-29T23:23:04.233226Z INFO Received block from consensus engine number=87337421 hash=0x65a9717839d06453f2309a736135ffa72e3cc4bf915711c8f45505e45a404f7e
2026-01-29T23:23:04.273012Z INFO Received block from consensus engine number=87337419 hash=0x4462e5cff78ed2d7da8fd17a89da6aa089f23eb1902ede24c975e5e2f4cf95cc
2026-01-29T23:23:04.273362Z INFO Received block from consensus engine number=87337425 hash=0x09a2b0b49a64b58c5609cc7924d5835e183543f48e99923a49c2344fdca19f9e
2026-01-29T23:23:04.273387Z INFO Received block from consensus engine number=87337426 hash=0xd74de93bb95b126cf2ab71d5b7fb053df4b7ca523b726c3cdf1887641e8bf126
2026-01-29T23:23:05.327495Z WARN Global signer not initialized for mining mode
2026-01-29T23:23:08.224563Z INFO Received block from consensus engine number=87337430 hash=0x676b702886f820e8b79296bda61719050ac0cbfa31a22f34e38c1f0cae5ede9c
2026-01-29T23:23:08.906801Z INFO Received block from consensus engine number=87337436 hash=0x49176547b40f4a548f9642b1a9785da5cb7a5be820d8a54563e851e379aac662
2026-01-29T23:23:08.910766Z INFO Received block from consensus engine number=87337432 hash=0xea8c4751fb968d7cd110002debb441095d544edaace801a530afdce6b469e005
2026-01-29T23:23:09.809980Z INFO Status connected_peers=9 stage=Headers checkpoint=87337373 target=0
2026-01-29T23:23:09.976123Z INFO Received block from consensus engine number=87337433 hash=0x9783e4651a68f41975656e22d35d774e0dcc24616d918534707cdcd336aacaad
2026-01-29T23:23:09.976187Z INFO Received block from consensus engine number=87337434 hash=0x913df702f5947b25e5f4a016f5071803c84964af8b6a3e9d6d48ccc1b5b67baf
2026-01-29T23:23:13.075773Z WARN Global signer not initialized for mining mode
2026-01-29T23:23:13.686254Z INFO Received block from consensus engine number=87337441 hash=0x5751fcba3e8cf6015ef6e7e56687e1e776ba5bbdff9c03992f774837207aa585
2026-01-29T23:23:13.686596Z INFO Received block from consensus engine number=87337439 hash=0x659271ae80254e775923f0ddd93ba7cdb758ea555d785e04eec4a9781e79e6ec
2026-01-29T23:23:13.686728Z INFO Received block from consensus engine number=87337443 hash=0x2dbcd95d5cd18dcc5d1bd15e5ad21c4519c0247a59ad339b1720ba452c3f28ca
2026-01-29T23:23:13.687027Z INFO Received block from consensus engine number=87337440 hash=0x721b2503869bf8ab8b3db74d6bc63eb59997268775ee2d23d605f219d62f370d
2026-01-29T23:23:14.870289Z INFO Received block from consensus engine number=87337446 hash=0x4b2a6b9bff64900bf175fa63c9d4e2efb3212c30c3cf230b24206a9569186cc8
2026-01-29T23:23:14.871300Z INFO Received block from consensus engine number=87337444 hash=0xf746e88a0df2aacc5d268a95097a51aba501c971c7d7c323c93d4273a7f552c2
2026-01-29T23:23:14.871610Z INFO Received block from consensus engine number=87337445 hash=0x40a75854c0910c94a7bdb8233a51e7f2fe91a454f7db2d706085db8bdb647030
2026-01-29T23:23:15.311996Z WARN Global signer not initialized for mining mode
2026-01-29T23:23:16.849246Z INFO Received block from consensus engine number=87337452 hash=0xd8153b9e253336d71c5111135c4609551212ae16391d522cb681a48bab111a71
2026-01-29T23:23:18.534356Z INFO Received block from consensus engine number=87337457 hash=0x62679dafaef37c90afedd9f8f93fd59cb3727487f0f9bf59e1ec59226b5ce0da
2026-01-29T23:23:18.821242Z INFO Received block from consensus engine number=87337450 hash=0xb1d70f1f5baae7078e0b93ecaf345c101a4a0a3c82994285c6801bd706cf61bf
2026-01-29T23:23:18.821890Z INFO Received block from consensus engine number=87337454 hash=0x9a95c47b89d545eeb112d3484c87fec0d271349a3f5e561837e621a97a829e3d
2026-01-29T23:23:18.822234Z INFO Received block from consensus engine number=87337458 hash=0xc0e33644874d6427b79ce91f0bfb49416138315c542707c3994489fb15d7d28d
Platform(s)
No response
Container Type
Kubernetes, Docker
What version/commit are you on?
ghcr.io/bnb-chain/reth-bsc:v0.0.7-beta
What database version are you on?
2026-01-30T10:19:34.599414Z INFO Initialized tracing, debug log directory: /root/.cache/reth/logs/bsc
Current database version: 2
Local database version: 2
Which chain / network are you on?
bsc testnet
What type of node are you running?
Archive (default)