Meta: HTTP Gateway cache control improvements

[lidel]

This is a meta issue for HTTP cache improvements that we should prioritize in go-ipfs:

• max-age on /ipns/ based on TTL – https://github.com/ipfs/go-ipfs/issues/1818
• dir-index-html on /ipfs/ should have some reasonable max-age (it won't change within releases)
• some HTTP header in /ipns/ responses which contains resolved IPNS (root CID) for faster cache invalidation of entire website when IPNS pointer changes
  • suggested in feat: Add X-IPFS-Root-CID header for faster cache invalidation #8617
    • proposed X-Ipfs-Roots in feat: X-Ipfs-Roots for smarter HTTP caches #8720 a way to indicate all CIDs required for resolving path segments from x-ipfs-path
• ensure we default to https://web.dev/stale-while-revalidate/ everywhere we can
• this maximizes utility of HTTP gateway – cache-control is used not only by end user (browsers), but also HTTP caching solutions, CDNs. We should have defaults that leverage HTTP caching tools and conversions, assuming people will manually configure HTTP caching on their own using some sane values is not realistic.

cc @thattommyhall & @mathew-cf) if there are more asks/ideas here

[thattommyhall]

Only thing I would add is a strategy of expiring at "top of the hour" or "end of the day" for the DirIndex pages makes dealing with when the etag does eventually change a bit nicer (you wont see different things on different servers for as long), which nudges me to prefer Expires to Cache-Control (but its not that much extra to calculate it anyway)

[mathew-cf]

An idea in addition to the ones proposed:
A public resolution endpoint for gateways could be useful as an inexpensive, cacheable call. There is no public endpoint (AFAIK) to verify resolutions for IPNS public keys or IPFS subpaths without fetching the whole file from a gateway, so an endpoint or query param exposing resolution could be helpful.

[thattommyhall]

@mathew-cf thats useful, but you'd have to rewrite to fetch /ipfs/<cid>/<path> or you couldnt be certain that the content you fetched matched. Like if I checked what the dnslink was for blog.ipfs.io then fetched /ipns/blog.ipfs.io it might have changed in the meantime

[mathew-cf]

By cacheable, I meant for a short TTL (~1 min). This resolution response can still be supplemented by DNS resolution for faster cache verification/invalidation. If you have the x-ipfs-roots header, you can check that the resolution and response match and choose whether to cache the response.

I'm thinking in the context of using Cloudflare workers here if that helps clarify the context.

[lidel]

@mathew-cf Thoughts on leveraging existing HTTP HEAD responses for these quick checks?

• They are inexpensive, don't return any payload, only headers, and do not require any new endpoint.

• HEAD request resolves the content path, and for unixfs DAGs fetch the bare minimum (root block of a file/directory) to get the size for Content-Length header.

• HEAD response could include additional metadata headers, for example:

  • let client know if the entire DAG is cached locally at the edge gateway, or if GET will require additional bitswap/fetch (enabling clients to decide which gateway to use for specific CID, prioritizing ones which have data in local cache etc)

[mathew-cf]

@lidel That works for me!