Bug: testnet node: no available backend

[morenonaitana]

I'm seeing the following error every day, it seems the backend is not stable, not sure if that is something currently being worked on but I prefer to create a ticket in case to rise the concern. Here is what I see:

 ERROR  [Balance] Failed to fetch TT2 bridge history: { address: '0xe364cc00c399930247ef70411f637c8e3f01549b',
  error:
   Error: {"jsonrpc":"2.0","id":22,"error":{"code":-900,"message":"no available backend"}}
       at checkRes (file:///Users/moreno/truflation/website/node_modules/@trufnetwork/kwil-js/dist-esm/api_client/client.mjs:245:13)
       at NodeKwil.callClient (file:///Users/moreno/truflation/website/node_modules/@trufnetwork/kwil-js/dist-esm/api_client/client.mjs:193:12)
       at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
       at async NodeKwil.baseCall (file:///Users/moreno/truflation/website/node_modules/@trufnetwork/kwil-js/dist-esm/client/kwil.mjs:340:24)
       at async NodeKwil.call (file:///Users/moreno/truflation/website/node_modules/@trufnetwork/kwil-js/dist-esm/client/node/nodeKwil.mjs:29:12)
       at async Object.handler (/Users/moreno/truflation/website/server/api/balance/truf-tt2-history.get.ts:82:1)
       at async file:///Users/moreno/truflation/website/node_modules/h3/dist/index.mjs:1962:19
       at async Object.callAsync (file:///Users/moreno/truflation/website/node_modules/unctx/dist/index.mjs:72:16)
       at async Server.toNodeHandle (file:///Users/moreno/truflation/website/node_modules/h3/dist/index.mjs:2249:7) }

I'm using those:
BILLING_TN_PROVIDER_HOODI=https://gateway.testnet.truf.network
BILLING_TN_CHAIN_ID_HOODI=testnet-v1

used this way:

const signerWallet = new Wallet(privateKey);

  cachedClient = new NodeTNClient({
    endpoint,
    signerInfo: {
      address: signerWallet.address,
      signer: signerWallet,
    },
    chainId,
    timeout: 30000,
  });

cc: @markholdex @vinarmani @MicBun

ETA: 16-Mar-2026

[MicBun]

Log: updated the instance size to t3a.large and tweaking memory on the server. Will keep monitoring now

[vinarmani]

@MicBun so you have identified this as a validator system resources issue?

[MicBun]

Yeah, one of it, the leader node keep crashing mid point and then automatically restarts. I previously set the automatic restart, but it is not enough, so I decided it is time to increase from t3.medium.

[morenonaitana]

@MicBun @markholdex can we set this task as priority? This is blocking us make progress with the prediction market as when this happens we can't get any data and test. Thank you!

[MicBun]

hi @morenonaitana, I will prioritize it again. While increasing the instance size helps, but seems not all fixed as I see sometimes the container restarted

[vinarmani]

log: I have added this to the backlog as part of the project

@MicBun this is a critical blocker for the migration to mainnet. It is important that the issue is actually identified and documented here in this issue, before we consider it closed again.

[MicBun]

Log:

1. The primary issue was ANTLR4 parser unbounded memory growth. It kills the node every 10 hours and recovery triggers.
2. I also keen to push things to the testnet server that requires restart. So it makes sense if it down repeatedly during my active time.

How the "No Available Backend" Error Occurs

The gateway (kgw) performs health checks every 6 seconds against the backend node via GET /api/v1/health/user. When the kwild process is OOM-killed:

1. The health check fails (connection refused)
2. The gateway moves the backend from available to unavailable
3. upstream.getBackend() returns nil
4. The gateway returns {"code":-900,"message":"no available backend"}
5. After docker restarts the container (~30-60s), health checks pass again
6. But the node may need additional time to catch up on blocks

Root Cause

The node runs out of memory and is killed by the operating system roughly every 10 hours.
The memory leak originates from the SQL parser. Every time the node processes a block, it executes several SQL operations for each of the 3 active ERC20 bridge instances (sepolia_bridge, hoodi_tt, hoodi_tt2). These SQL operations go through an internal parser (ANTLR4) that maintains a global cache to speed up repeated parsing. However, this cache grows indefinitely and is never cleaned up, eventually consuming all available memory.

In simple terms: the node remembers every SQL pattern it has ever seen, never forgets any of them, and eventually runs out of room.

[MicBun]

@holdex issue set-eta 16-Mar-2026