Revamp EstuaryV1 to use Delta/EV2

[alvin-reyes]

Proposal

I wanted to propose that we completely revamp EstuaryV1 to use Delta as the deal making platform. There are a few things that needs to be considered but for the most part, it should work.

Considerations

• use whypfs-core as the node implementation.
• remove the splitting and introduce an estuary chunker as a separate component. This component will consume 32GB and above files. The component will car the files and upload them straight to delta.
• any files below 32GB but more than 250B will be dealt with using the miners from data.storage.market. Delta will take care of the deal making, retries and replication.
• estuary-www will have to strip / modify the staging area. The staging area will be for files that are larger than 32GB. Instead of zoning, it'll be IPLD root node with the links (from generating the CAR for the file).
• estuary-www will need to be connected to several new endpoints -
  • 1. estuary-auth,
  • 2. delta api nodes (node.delta.store/cake.delta.store)
  • 3. delta-chunker endpoint
  • 4. API node (we will be using this until we have all the functions done).
• estuary-www will have a new option to connect to delta node.
• estuary-www will need to use the stats endpoint to get the status of each deal made.

New components

• Estuary chunker (based on my delta-chunker that uses delta car).
• whypfs-core, estuary-auth, delta node.

How is this different from the current EstuaryV1?

• Delta doesn't do aggregation or consolidation. The amount of resource we are saving from removing those processes are huge.
• Estuary node will be a gateway instead - serving hot retrievals from its node.
• Delta node(s) will be dedicated and all deal-making processes are done on a separate compute resource.