ENSApi: Persist ENSIndexerPublicConfig to the database

[shrugs]

Background

Currently:

• ENSIndexer exposes an API for indexing status and for its config
• ENSApi directly sends API calls to ENSIndexer to get its indexing status and config
• ENSApi uses SWR caching to remember the latest indexing status and config from ENSIndexer, such that ENSApi can survive crashes of ENSIndexer if and only if the ENSApi instance was able to successfully fetch these values from ENSIndexer during the lifecycle of the ENSApi process instance.
• ... Therefore: if ENSIndexer is crashed and ENSApi is restarted / launched, the new ENSApi currently will also fail to start. This would be a bad situation and we want to ensure it can't impact ENSNode deployments.

Goals

1. Ensure ENSAPI can survive situations where ENSIndexer is down and ENSApi is restarted.
2. Update ENSApi and ENSIndexer so that they stop exchanging the config and indexing status via API calls to ENSIndexer and instead they exchange this state exclusively through ENSDb. Here, ENSIndexer should push this state into ENSDb, while ENSApi should pull this state from ENSDb.
3. Unblock the next phase of advancements on ENSAnalytics. More specifically, as a separate future goal, we want to transition the ens-referrals package out of the ENSNode repo and into the ens-referrals repo (which would be converted into a monorepo). Then, in the ens-referrals repo, we would want to create a new backend service for the leaderboard APIs that would be the first example of a custom app that would build on top of ENSDb and expose a tailored API built specifically for the needs of that app. This would allow us to transition all of the ENSAnalytics out of ENSNode and into this new custom app in the ens-referrals repo. Before we take all of these actions though it will be important that we first action the goals described in this issue so that there is a mature mechanism for an app to build custom APIs on top of ENSDb and to integrate with the indexing status and config stored in ENSDb. Related issue: Transition ENSAnalytics out of ENSApi #1352
4. Protect ENSNode operators against a scenario where they have all ENSNode services up and running. Then imagine they keep ENSApi up and running while terminating ENSIndexer, wiping the configured DATABASE_SCHEMA, and then starting a new ENSIndexer with config changes that are incompatible with the config that has been cached into memory by ENSApi. The logic proposed here aims to ensure that a max of 1 minute might exist where an ENSApi instance is attached to an incompatible DATABASE_SCHEMA without knowing it.
5. Create a new ENSDbClient class that encapsulates logic for the following in a way that would be nicely reusable across apps building on top of ENSDb, including ENSApi and the future app described above in goal 3.
   1. Manages SWRCache that read both the config and indexing status from a DATABASE_SCHEMA in ENSDb.
   2. Enforcing that the indexing status and config objects that it caches are compatible. Ex: the chain ids perfectly align. Etc.
   3. Provides a helper function such as isHealthy that gives a simple mechanism to identify if both a config and indexing status are held in memory and compatible with each other.
6. Improve the startup sequence of ENSApi such that it doesn't crash itself just because it is waiting on ENSIndexer to successfully initialize itself. Instead, it should patiently wait. Related issue: ENSApi & ENSIndexer re-try mechanism for long starting dependency (Render) #1303
   1. Update the logic for the /health endpoint in ENSApi such that it considers itself "unhealthy" if the ENSDbClient.isHealthy is false.

Non-Goals

1. We should not remove the config or indexing status APIs from ENSIndexer. I believe all of our current usage of these APIs should be removed as a result of the changes described here, but I think it's best to keep these APIs living in ENSIndexer anyway as maybe there will be some special reason for them in the future that we cannot predict now.

Goals

1. ENSIndexer
   1. Each time it starts it should:
      1. Check if a config has already been saved into the configured DATABASE_SCHEMA. If so, read it from the DATABASE_SCHEMA and verify that it is compatible with the config object it has built in memory. If it is not compatible, refuse to start and provide a meaningful error message.
      2. If no config has been saved into the configured DATABASE_SCHEMA yet, save it there.
   2. At a frequent time interval (ex: maybe once per second?):
      1. Build a new indexing status snapshot in memory. If this fails for some reason, log the error and do nothing for this iteration of the asynchronous background loop.
      2. Is the omnichain indexing status in that snapshot OmnichainIndexingStatusSnapshotUnstarted? If so, log the error and do nothing for this iteration of the asynchronous background loop.
      3. Upsert the indexing status snapshot into the DATABASE_SCHEMA.
2. ENSApi
   1. Update our logic for SWRCache so we update the signature of the fn such that it optionally receives a copy of whatever the latest cached value is as input. (Optionally meaning that if no value is already cached it receives undefined, but if a value is already cached it receives that value). This way the fn can internally decide special logic it may want to perform as a function of whatever the already cached value is. The purpose of this idea will be explained below.
   2. Each time ENSApi starts it should:
      1. Build a SWRCache for reading the config from the configured DATABASE_SCHEMA. This should proactively initialize itself and proactively revalidate itself once per minute.
         1. The fn should work to read the config from the configured DATABASE_SCHEMA and then validate / deserialize it. Anytime this fetch or validation / deserialization fails, log the error.
         2. The fn should make use of the idea described above for the fn to optionally receive a copy of whatever the latest cached value as input was. More specifically, each time a new config value is built in memory (not yet returned from fn) this function should compare that the config has not changed in a way that would be incompatible. In other words, changing a RPC endpoint is an example of a compatible change, however, changing the indexed ENS namespace is an example of an incompatible change. All transitions from no cached result -> cached result would pass this check, this check is purely for validating the transition from cached result A -> cached result B. If a config change is incompatible, ENSApi should forcefully terminate itself (open to other suggestions).
      2. Build a SWRCache for the latest indexing status snapshot. But now the fn associated with this should work to read the indexing status snapshot from the DATABASE_SCHEMA rather than reading the latest indexing status snapshot from direct API calls to ENSApi. If no indexing status snapshot is found in the DATABASE_SCHEMA then log an error. If it is found, validate it. If validation / deserialization fails, log an error. If it is found and all validation / deserialization succeeds, compare the indexing status against the config as read from the SWRCache that manages the config. If the config is not cached yet, log a message. If the config is cached, validate that the new indexing status is compatible with the config. For example, all the chain ids should match up perfectly. Etc. If the new indexing status is compatible, successfully cached it.
      3. store the indexing status in a proactively revalidating SWRCache. need to persist/cache the ensindexerpublicconfig to database so ensapi can survive restarts without ensindexer availability
      4. Consider itself "unhealthy" until both its SWRCacheIf ENSApi it is not found log an error and terminate. If it is found, validate it. If validation / deserialization fails, log an error and terminate. If it is found and all validation / deserialization succeeds, log the config it will be using to the console and store the cached config in memory indefinitely for the remaining lifecycle of the ENSApi process instance.
   3. Remove the following environment variables from ENSApi: ENSINDEXER_URL

[lightwalker-eth]

@tk-o While working on this issue, I think there may also be an opportunity to advance this goal too. Appreciate your advice if we include this goal in this issue too or come back to it separately: #1227

[tk-o]

For ENSApi, there must a validation step added, such that whenever ENSApi wants to begin operations on a DATABASE_SCHEMA through an ENSDbClient, the client always begins by querying all the key->value pairs in this "ensnode_metadata" table. If any of the following error cases occur, the ENSDbClient can give nice error messages, for example we could detect error cases including:

1. Cannot connect to database (at all)
2. DATABASE_SCHEMA not found
3. Invalid DATABASE_SCHEMA table structure, for the case that the "metadata" table could not be found.
4. Unsupported ENSDb version, for the case that the ensdb_version value doesn't match the value expected by the client.