feat: add HyperSync provider for faster event indexing

[wa0x6e]

Summary

Toward https://github.com/snapshot-labs/workflow/issues/787

This PR will add a new HyperSync EVM provider, to use HyperSync (https://envio.dev/#hypersync) as events discovering.

HyperSync is able to return a list of paginated events, for the whole chain, compared to the RPC flow where we had to search for events in the range of 1000 blocks at a time.

This speed up the events searching, down from few weeks to quasi instantaneous.

The whole remaining indexing time is now due to read/write speed on the database layer (psql).

You can create a free api token, using the free plan. The free plan limits is amply enough to index arbitrum.

Test plan

In your sx-monorepo app, link your checkpoint dependency to this PR, either by using bun link, or editing your package.json to point to your local copy using file:/ instead of just the version number.

Don't forget to run yarn build on checkpoint.

in sx, apps/api/src/evm/index.ts, edit the arb indexer to use

const arb1Indexer = new evm.HyperSyncEvmIndexer(createWriters(arb1Config), {
  apiToken: process.env.HYPERSYNC_API_TOKEN || ''
});

Add your api token to your apps/api/.env file

Uncomment the arbitrum contracts (keep GMX commented due to issue with timestamp parsing)

Start the sx app with only arbitrum

ENABLED_NETWORKS=arb1 bun run dev

In the logs, you should see the "preloading" action only once between each events processing, the indexer do not have to find the events by searching blocks by blocks

[22:42:10.543] INFO: preloading blocks
    component: "container"
    indexer: "arb1"
    start: 157825417
    end: 451402859

Start the sx app with eth

ENABLED_NETWORKS=eth bun run dev

In the logs, you should see multiple "preloading" events back to back when events are not found

[22:58:41.798] INFO: preloading blocks
    component: "container"
    indexer: "eth"
    start: 12006099
    end: 12007099
[22:58:42.516] INFO: preloading blocks
    component: "container"
    indexer: "eth"
    start: 12007100
    end: 12008200
[22:58:43.340] INFO: preloading blocks
    component: "container"
    indexer: "eth"
    start: 12008201
    end: 12009401
``

[Sekhmet]

I think it would be better if we just have two variants of indexers.

EvmProvider and HyperSyncEvmProvider (and HyperSyncEvmIndexer to match). I think it would be better that hypersync config is not part of main config of Checkpoint as providers/indexers are meant to be independent so it can just be passed when doing new evm.HyperSyncEvmIndexer(writers, { hyperSyncApiKey: "ASDF" }).

HyperSyncEvmProvider could just extend EvmProvider. To avoid duplication we could add two methods in EvmProvider: getBlock and scanEvents that HyperSyncEvmProvider would just override.

[wa0x6e]

I think it would be better if we just have two variants of indexers.

EvmProvider and HyperSyncEvmProvider (and HyperSyncEvmIndexer to match). I think it would be better that hypersync config is not part of main config of Checkpoint as providers/indexers are meant to be independent so it can just be passed when doing new evm.HyperSyncEvmIndexer(writers, { hyperSyncApiKey: "ASDF" }).

HyperSyncEvmProvider could just extend EvmProvider. To avoid duplication we could add two methods in EvmProvider: getBlock and scanEvents that HyperSyncEvmProvider would just override.

How will you pass the API key from sx apps/api to checkpoint ? Using env var ?

The setup I am thinking of is to have 2 completely independent providers, and let the container.ts decide which one to use (one for live, the other for batch), so hypersync can function 100% without rpc provider.

[Sekhmet]

https://github.com/snapshot-labs/sx-monorepo/blob/d5eb6885a6eb531ec8d703fd735c3dbb6aa56db8/apps/api/src/evm/index.ts?plain=1#L70

This would basically become:

const arb1Indexer = new evm.HyperSyncEvmIndexer(createWriters(arb1Config), { hyperSyncApiKey: process.env.HYPER_SYNC_API_KEY });

[bonustrack]

Should be a single word for HyperSync filename, its like WalletConnect, we wouldnt do wallet-connect-provider.ts

Suggested change

	import { HyperSyncEvmProvider } from './hyper-sync-provider';
	import { HyperSyncEvmProvider } from './hypersync-provider';

[wa0x6e]

Should we use Hypersync all lowercase too, to match?

[bonustrack]

No

[bonustrack]

Why we need to change this file and base provider? It would be easier to review if we don't need to changes EvmProvider

[wa0x6e]

To avoid code duplication, and for proper class inheritance.