lorre cannot work:Could not write blocks or accounts checkpoints to the database.

[sukicyh]

lorre suddenly cannot work

error log is as below

2022.04.07 03:53:17 [lorre-tezos-indexer-akka.actor.default-dispatcher-11] ERROR TokenContracts  (tech.cryptonomic.conseil.indexer.tezos.michelson.contracts.TokenContracts.mapIdsMatch:171) - A token balance update was found where the map of the given token is not yet identified from contract origination
 map_id: 36
 token: ContractId(KT1LN4LPSqTMS7Sd2CJw4bbDGRkMv2t68Fy9)

2022.04.07 03:53:17 [lorre-tezos-indexer-akka.actor.default-dispatcher-11] ERROR TokenContracts  (tech.cryptonomic.conseil.indexer.tezos.michelson.contracts.TokenContracts.mapIdsMatch:171) - A token balance update was found where the map of the given token is not yet identified from contract origination
 map_id: 36
 token: ContractId(KT1LN4LPSqTMS7Sd2CJw4bbDGRkMv2t68Fy9)

2022.04.07 03:54:08 [lorre-tezos-indexer-akka.actor.default-dispatcher-17] ERROR TezosIndexer    (tech.cryptonomic.conseil.indexer.tezos.TezosIndexer.processTezosBlocks:288) - Could not fetch blocks from client
io.circe.DecodingFailure$$anon$2: Attempt to decode value on failed cursor: DownField(metadata)


2022.04.07 03:54:08 [main] INFO  t.c.c.i.Lorre   (tech.cryptonomic.conseil.indexer.Lorre.shutdown:83) - Doing clean-up.

2022.04.07 03:54:08 [lorre-tezos-indexer-akka.actor.default-dispatcher-9] ERROR BlocksProcessor (tech.cryptonomic.conseil.indexer.tezos.processing.BlocksProcessor.applyOrElse:54) - Could not write blocks or accounts checkpoints to the database.
java.util.concurrent.RejectedExecutionException: Task slick.basic.BasicBackend$DatabaseDef$$anon$3@56c8fa12 rejected from slick.util.AsyncExecutor$$anon$1$$anon$2@697f6a15[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 12122]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
        at slick.util.AsyncExecutor$$anon$1$$anon$4.execute(AsyncExecutor.scala:161)
        at slick.basic.BasicBackend$DatabaseDef.runSynchronousDatabaseAction(BasicBackend.scala:265)
        at slick.basic.BasicBackend$DatabaseDef.runSynchronousDatabaseAction$(BasicBackend.scala:263)
        at slick.jdbc.JdbcBackend$DatabaseDef.runSynchronousDatabaseAction(JdbcBackend.scala:37)
        at slick.basic.BasicBackend$DatabaseDef.slick$basic$BasicBackend$DatabaseDef$$runInContextInline(BasicBackend.scala:242)
        at slick.basic.BasicBackend$DatabaseDef.runInContextSafe(BasicBackend.scala:148)
        at slick.basic.BasicBackend$DatabaseDef.runInContext(BasicBackend.scala:142)
        at slick.basic.BasicBackend$DatabaseDef.runInContext$(BasicBackend.scala:141)
        at slick.jdbc.JdbcBackend$DatabaseDef.runInContext(JdbcBackend.scala:37)
        at slick.basic.BasicBackend$DatabaseDef.$anonfun$runInContextInline$1(BasicBackend.scala:172)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
        at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)


2022.04.07 03:54:08 [main] INFO  HikariDataSource (com.zaxxer.hikari.HikariDataSource) -  - Shutdown initiated...

2022.04.07 03:54:08 [main] INFO  HikariDataSource (com.zaxxer.hikari.HikariDataSource) -  - Shutdown completed.

Exception in thread "main" java.util.concurrent.TimeoutException: Futures timed out after [10 seconds]
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259)
        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263)
        at scala.concurrent.Await$.$anonfun$result$1(package.scala:220)
        at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:57)
        at scala.concurrent.Await$.result(package.scala:146)
        at tech.cryptonomic.conseil.indexer.Lorre$.shutdown(Lorre.scala:84)
        at tech.cryptonomic.conseil.indexer.Lorre$.delayedEndpoint$tech$cryptonomic$conseil$indexer$Lorre$1(Lorre.scala:80)
        at tech.cryptonomic.conseil.indexer.Lorre$delayedInit$body.apply(Lorre.scala:21)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1$adapted(App.scala:80)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at scala.App.main(App.scala:80)
        at scala.App.main$(App.scala:78)
        at tech.cryptonomic.conseil.indexer.Lorre$.main(Lorre.scala:21)
        at tech.cryptonomic.conseil.indexer.Lorre.main(Lorre.scala)
2022.04.07 03:54:35 [shutdownHook1] INFO  t.c.c.i.Lorre   (tech.cryptonomic.conseil.indexer.Lorre.shutdown:83) - Doing clean-up.

Exception in thread "shutdownHook1" java.util.concurrent.TimeoutException: Futures timed out after [10 seconds]
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259)
        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263)
        at scala.concurrent.Await$.$anonfun$result$1(package.scala:220)
        at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:57)
        at scala.concurrent.Await$.result(package.scala:146)
        at tech.cryptonomic.conseil.indexer.Lorre$.shutdown(Lorre.scala:84)
        at tech.cryptonomic.conseil.indexer.Lorre$.$anonfun$new$2(Lorre.scala:45)
        at scala.sys.ShutdownHookThread$$anon$1.run(ShutdownHookThread.scala:37)

[piotrkosecki]

Hello!
After our investigation it looks like tezos nodes 12.1 and 12.2 are not always returning metadata. Because of that Lorre breaks. The fix is to use node with --metadata-size-limit=unlimited parameter, but unfortunately you need to resync your node as metadata which is missing cannot be recovered.

[sukicyh]

@piotrkosecki resync my tezos node? add that flag to the lorre?
thank you!

[piotrkosecki]

you need to add this flag to the tezos node

[sukicyh]

@piotrkosecki and resync tezos node?

[piotrkosecki]

we are currently testing 12.3 node and so fat it looks good, in case 12.1/12.2 you need to resync tezos node.