solana indexer hang Mar 19- Mar 24, 2026, likely stream getting stuck in run_stream

[ppca]

Conclusion first: not the sol indexer is stuck, but the stream in run_stream is stuck.

Observation:
This only happened on node 11 and node 4.
I looked into the solana indexer hang on Mar 19 on node 11, the grafana dashboard shows the number of solana requests and block height both stuck
But I checked our gcp logs, it is clear that node 11 found solana requests consistently during the time window shown as stuck in grafana dashboard.
I looked into our codebase, we emit the latest block height metric only when a block event is emitted from the stream(code). And the number of requests metric is coming from multichain_sign_request_latency_sec, which only gets emitted in record_request_latency, and this won't get emitted if a signature generator has not been started.

So my guess of what's happening is: not the sol indexer is stuck, but the stream in run_stream is stuck.

Fix:
Not sure why the stream can get stuck yet. ideas? @volovyks @jakmeier @ChaoticTempest

[jakmeier]

One possibility is that the MessageInbox as a whole got stuck, as @ChaoticTempest pointed out. In that case, the SignatureSpawner will also get blocked and will no longer poll on sign_rx. That in turn fills up the sign_tx -> sign_rx channel (capacity is 16k messages) . Then run_stream would get stuck on the next process_sign_event calling sign_tx.send().

We should be able to see where exactly this is a problem once we have the metrics from with #701

More generally, incoming messages from indexers are unlimited. But channels to send out messages to other tasks are all bound. This is fundamentally an unstable design that we need to resolve at some point. (More details in #731)

[jakmeier]

FWIW, my experimentation (see this draft PR #739) suggests we can get stuck in other ways. At least so far, I couldn't replicate any channel buffers filling up, even when reducing their sizes to 1 and hammering it with 1000s of requests.