ENSRainbow client graceful retry mechanism

[tk-o]

@lightwalker-eth suggested:

Add a more graceful retry mechanism for the case that there's a client network error here. Perhaps we would retry a few times with some backoff before finally giving up? Each failure should write a warning to the logs

[tk-o]

I think this one would be a good candidate for the ensnode-utils package, so that ENSNode could have ability to wrap some code around and retry it (+ attach logs).

[tk-o]

Connection reset error would be another candidate to handle. Just spotted this error in ENSNode logs:

4:43:53 AM ERROR indexing   Error while processing '/eth/Registry:NewOwner' event in '1' block 14746518
TypeError: fetch failed
    at Module.labelByHash (/app/apps/ensnode/src/lib/graphnode-helpers.ts:29:24)
    at /app/apps/ensnode/src/handlers/Registry.ts:133:31
  27 |  **/
  28 | export async function labelByHash(labelhash: Labelhash): Promise<string | null> {
> 29 |   const healResponse = await ensRainbowApiClient.heal(labelhash);
     |                        ^
  30 |
  31 |   if (healResponse.status === StatusCode.Success) {
  32 |     // original label found for the labelhash
caused by: Error: read ECONNRESET
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:216:20)
Event arguments:
  node    0x93cdeb708b7545dc668eb9280176169d1c33cfd8ed6f04690a0bcc88a93fc4ae
  label   0x9c58a488c776266339a3f971a3a3c419dab825d2a6887dd42f2522680038cd4f
  owner   0x283Af0B28c62C092C9727F1Ee09c02CA627EB7F5
Log:
  index     141
  address   0x00000000000C2E074eC69A0dFb2997BA6C7d2e1e
Transaction:
  hash   0xaa5ce46d5ac4718cac5de79fa5e4ff223ec7e83ddb54b12f62ea7ba007d5d044
  from   0x7C947AF000D8F848624C8b97DBA05b1DaC50b32f
  to     0x283Af0B28c62C092C9727F1Ee09c02CA627EB7F5
Block:
  hash        0x7f29101f6507d28906aeeca7705429cf29ff92f5e394e759d440c202708330a3
  number      14746518
  timestamp   1652156648
4:43:53 AM WARN  process    Encountered indexing error, starting shutdown sequence
/app/apps/ensnode:
 ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  ensnode@0.0.1 start: `ponder start "--schema" "6f7ba2e5-f224-41dd-a38f-56a3464c2fd8"`
Exit status 1

[tk-o]

Plus another error kind to handle: SocketError: other side closed.

3:06:02 AM ERROR indexing   Error while processing '/eth/Registry:NewOwner' event in '1' block 14703519
TypeError: fetch failed
    at Module.labelByHash (/app/apps/ensnode/src/lib/graphnode-helpers.ts:29:24)
    at /app/apps/ensnode/src/handlers/Registry.ts:133:31
  27 |  **/
  28 | export async function labelByHash(labelhash: Labelhash): Promise<string | null> {
> 29 |   const healResponse = await ensRainbowApiClient.heal(labelhash);
     |                        ^
  30 |
  31 |   if (healResponse.status === StatusCode.Success) {
  32 |     // original label found for the labelhash
caused by: SocketError: other side closed
    at TLSSocket.<anonymous> (node:internal/deps/undici/undici:6238:28)
    at TLSSocket.emit (node:events:530:35)
    at endReadableNT (node:internal/streams/readable:1698:12)
    at processTicksAndRejections (node:internal/process/task_queues:90:21)