[IPFS KPIs Update] Milestone 4: Gateways Performance Measurements

[yiannisbot]

Following up from the discussion and requests at: #176, ProbeLab will be working to update IPFS KPIs presented at https://probelab.io/. Work includes several milestones, which will be captured in separate issues in this repo. This issue captures Milestone 4 that includes the following Tasks.

Description: In this milestone, we will measure the content retrieval performance via the traditional IPFS Gateways, trustless HTTP gateways, and service worker gateways. The content that we will retrieve will be: 1) seeded by the bitswap traffic and 2) our controlled nodes. This is because we cannot be sure that the CIDs we learn via bitswap actually exist while in the controlled setup we will have that guarantee. The retrieval will happen through our controlled nodes that are deployed in multiple AWS regions All of the tasks below involve the publication of relevant plots on probelab.io.

Segment the measurements by:

1. Gateway type: Traditional, Trustless, Service Worker. Sub-segment the traditional and trustless retrievals by cached vs uncached (only possible to reliably determine with controlled content)
2. Seed source: Bitswap, controlled
3. Retrieval Location: AWS Region

Task 1: HTTP Retrievals via Traditional & Trustless Gateways. Set up probes to measure the HTTP retrieval through traditional and trustless IPFS gateways from different AWS regions.
Task 2: Service Worker Gateway. Set up a headless chrome instance to measure retrieval performance through a service worker gateway, making sure that gateway resources are measured separately to the actual IPFS performance. See work initiated already here: probe-lab/tiros#14.

[dennis-tra]

Hey everyone,

With the upcoming changes of deprecating the regular ipfs.io/dweb.link gateways and instead fully relying on the SW implementation I want to make sure to not build anything that becomes obsolete in a few months.

As stated above, we promised to measure the retrieval performance from different geographic locations of ...

• the traditional HTTP Gateways
• trustless HTTP Gateways
• SW Gateways

The CIDs which we will request will come from our bitswap-sniffer tool plus a controlled set of CIDs that we host.

The measurement setup will consist of headless chrome instances running in different AWS regions which we will orchestrate to retrieve CIDs through any of the gateway implementations.

For the SW Gateway, as suggested by @lidel, the current plan is to parse the log messages to extract exact timing information about different steps of the retrieval process (time to until we found the first provider, ttfb, etc.). This actually sounds quite brittle to me as it could break at any time in the future when things get refactored or if the message is just slightly changed. But if it's the only way to get internal information then it is what it is I guess.

@achingbrain:

• Do you have an alternative idea to perform this kind of blackbox testing while extracting internal information?
• If log parsing is still the way to got, should I mark the messages in the code?

@ns4plabs:

• The traditional ipfs.io/dweb.link HTTP gateways will be shut down sometime next year if I'm not mistaken. I'm still planning to measure retrieval performance because this could then also be applied to potential other public gateways. What is the current caching policy? (mainly, how long is content cached at the different tiers)

@lidel:

• @lidel we promised to measure traditional and trustless HTTP gateway retrieval but I don't quite see that this adds value. It's just that the gateway would return raw blocks instead of the assembled file, right? I actually don't think it's worth separately measuring that. Wdyt? Is the assembly step a heavy process that could significantly impact retrieval performance? I doubt it, but would like to hear your opinion.

Thanks everyone!

[achingbrain]

Do you have an alternative idea to perform this kind of blackbox testing while extracting internal information?

Yes - an onProgress option can be passed to @helia/verified-fetch, this is a callback that is invoked with event objects when certain things happen during an operation.

There are a number of events already implemented, more can be added as needed.

The service worker can export these events somehow - send them to an external service or otherwise make them available.

[joshdougall]

Hello! ChainSafe is in the process of launching a global gateway, that also has regional endpoints available for explicit region selection. This gateway is called Orbitor, and the gateway endpoint is available at ipfs.orbitor.dev. As Probelab begins to monitor gateway performance and metric tracking across these gateways, I am commenting here to request that the ChainSafe endpoints be added to the list of monitored gateways. It would be great to drive more monitoring datapoints and compare to IPFS.io and dweb.link gateway performance. The endpoints to be made available are as follows:

Global endpoint:

• ipfs.orbitor.dev

Regional endpoints:

• latam.orbitor.dev
• apac.orbitor.dev
• eu.orbitor.dev

[dennis-tra]

Hi everyone,

I just wanted to give a signal that this is now my priority for this and next week 👍 I'll circle back here with questions or as soon as I have something to show :)

[dennis-tra]

Hi @joshdougall, I'm getting a 403 for any ipfs.orbitor.dev CID. Is there authentication required?

[hamidmuslih]

no auth layer is setup,
@dennis-tra can you please paste the complete URL you are accessing?

below is a working CID on my side
https://ipfs.orbitor.dev/ipfs/bafkreie7ohywtosou76tasm7j63yigtzxe7d5zqus4zu3j6oltvgtibeom

[dennis-tra]

Here's an example:

curl -vv https://ipfs.orbitor.dev/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

it resolves in the browser but this is the curl output:

$ curl -vv https://ipfs.orbitor.dev/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi
* Host ipfs.orbitor.dev:443 was resolved.
* IPv6: 2606:4700:20::681a:705, 2606:4700:20::681a:605, 2606:4700:20::ac43:48a3
* IPv4: 104.26.7.5, 172.67.72.163, 104.26.6.5
*   Trying 104.26.7.5:443...
* Connected to ipfs.orbitor.dev (104.26.7.5) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=orbitor.dev
*  start date: Dec  4 16:04:07 2025 GMT
*  expire date: Mar  4 17:03:57 2026 GMT
*  subjectAltName: host "ipfs.orbitor.dev" matched cert's "*.orbitor.dev"
*  issuer: C=US; O=Google Trust Services; CN=WE1
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://ipfs.orbitor.dev/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: ipfs.orbitor.dev]
* [HTTP/2] [1] [:path: /ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi HTTP/2
> Host: ipfs.orbitor.dev
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 403
< date: Tue, 13 Jan 2026 13:36:50 GMT
< content-type: text/html; charset=UTF-8
< content-length: 7175
< accept-ch: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA
< cf-mitigated: challenge
< critical-ch: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA
< cross-origin-embedder-policy: require-corp
< cross-origin-opener-policy: same-origin
< cross-origin-resource-policy: same-origin
< origin-agent-cluster: ?1
< permissions-policy: accelerometer=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
< referrer-policy: same-origin
< server-timing: chlray;desc="9bd548ea3a3e8a82"
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=v%2B5y2ResElAb%2FFSRfpeREONwgiElOCevTfcDo66gkj3QzG%2FFhL66W5iqcGYd%2Bi1R2NGWqYDIND0t1rlDWYC4S8xrCwP7a%2F2qHEkI9G%2Fcu7A%2B5RRlmRNFGaT0xVjcQx9cYTQ%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 9bd548ea3a3e8a82-TXL
<
...

Note the line < HTTP/2 403

[hamidmuslih]

Ah, noted. We have enabled bot protection on our endpoint to prevent misuse of the IPFS gateway. As a result, Cloudflare’s security layer treats curl requests as bot traffic.

[hamidmuslih]

@dennis-tra I have relaxed the bot protection, now it should be fine

[dennis-tra]

An update for the above.

I have set up measurement infrastructure to probe the traditional HTTP Gateways and the Service Worker gateways in the following four AWS regions: eu-central-1, ap-southeast-2, us-east-2, and us-west-1.

Each deployment probes controlled and organic CIDs. The controlled CIDs are hosted on a controlled Kubo node which is running in us-east-1 and hosts 4k CIDs of 1kB files of random data. That Kubo node is also accessible via WSS to theoretically allow the service worker gateway to retrieve content directly from it. The organic CIDs come from our Bitswap traffic sniffer which
constantly collects CIDs that are requested in the network.

The HTTP and Service Worker deployments constantly try to request controlled and organic CIDs from the gateways/network with a ratio of 1:4. 20% or requests are made for controlled CIDs and 80% are made for organic CIDs. The reason is that I assume our controlled CIDs to be stably accessible while requests for organic CIDs probably fail more often. Therefore, I introduced this ratio to get a large enough sample size for both.

Currently, we probe the following HTTP gateways:

dweb.link
ipfs.filebase.io
ipfs.io
w3s.link
ipfs.orbitor.dev

and only the inbrowser.link service worker gateway (which offloads requests to trustless-gateway.link).

Each HTTP gateway is probed twice for the same CID to measure the uncached and cached retrieval performance. We also probe the trustful (?) and trustless retrieval performance by adding/omitting the format=car query parameter to the request.

A data point for an HTTP gateway probe contains the following information:

region:              us-west-1
tiros_version:       284f3680+dirty
gateway:             ipfs.io
cid:                 QmcD6paKVWoN81QZZ7ftuxMura79GHHvWTxBT5fxWzufsk
cid_source:          bitsniffer_bitswap
format:              none // we're not requesting a raw or car file
request_start:       2026-01-13 19:16:23.970
dns_duration_s:      0.000663393
conn_duration_s:     0.003727872
ttfb_s:              0.039841951
download_duration_s: 0.040022619 // (we limit the download to 10MiB)
bytes_received:      8895
content_length:      8895
download_speed_mbps: 1.6956277737713452
status_code:         200
ipfs_path:           /ipfs/QmcD6paKVWoN81QZZ7ftuxMura79GHHvWTxBT5fxWzufsk
ipfs_roots:          QmcD6paKVWoN81QZZ7ftuxMura79GHHvWTxBT5fxWzufsk
cache_status:        HIT // (or MISS)
content_type:        image/jpeg
car_validated:       ᴺᵁᴸᴸ
redirect_count:      0
final_url:           https://ipfs.io/ipfs/QmcD6paKVWoN81QZZ7ftuxMura79GHHvWTxBT5fxWzufsk
error:               ᴺᵁᴸᴸ
created_at:          2026-01-13 19:16:24.011

A data point for a service worker gateway probe contains the following information:

region:                   eu-central-1
tiros_version:            a2142d4b+dirty
gateway:                  inbrowser.link
cid:                      bafkreihpidrietyqj2jurnjj3cbc7qpbe2bsdpzsbh63c3r5zqu3zw26va
cid_source:               bitsniffer_bitswap
url:                      https://inbrowser.link/ipfs/bafkreihpidrietyqj2jurnjj3cbc7qpbe2bsdpzsbh63c3r5zqu3zw26va
total_ttfb_s:             4.818421
final_ttfb_s:             1.9619170000000001
time_to_final_redirect_s: 2.856504
service_worker_version:   @helia/service-worker-gateway/2.2.0#production@2802fd7
status_code:              200
content_type:             application/octet-stream
content_length:           82799
ipfs_path:                /ipfs/bafkreihpidrietyqj2jurnjj3cbc7qpbe2bsdpzsbh63c3r5zqu3zw26va
ipfs_roots:               bafkreihpidrietyqj2jurnjj3cbc7qpbe2bsdpzsbh63c3r5zqu3zw26va
server_timings:           {"ipfs_resolve":"0"}
found_providers:          -1
served_from_gateway:      true
delegated_router_ttfb_s:  ᴺᵁᴸᴸ
trustless_gateway_ttfb_s: 1.8158020000000001
error:                    ᴺᵁᴸᴸ
created_at:               2026-01-26 12:08:52.000

• total_ttfb_s - Time to first byte including all redirects (seconds, browser ResourceTiming)
• final_ttfb_s - Time to first byte of final service worker response only
• time_to_final_redirect_s - Time from initial request to final service worker request
• delegated_router_ttfb_s - Fastest TTFB from any delegated router request (null if aborted)
• trustless_gateway_ttfb_s - Fastest TTFB from any successful trustless gateway request

Technical Note

Requesting data with the service worker gateway results in a set of redirects which made it challenging to match on the final content-retrieving redirect. This is necessary to take accurate timing measurements. We had discussed a way to match on this final request which didn't work and now I'm detecting the final request if the response headers contain the x-ipfs-path or x-ipfs-roots fields. However, these two are only there if the request was successful so I'm also checking if the access-control-expose-headers is present and contains x-ipfs-path or x-ipfs-roots. That header is always there on the final response and I believe won't break unexpectedly.

Next steps

• I'll observe the deployment, adjust probing frequency and iron out some kinks (because I think we're oversampling right now)
• Add e2e tests to the Tiros repository
• Visualize the data on probelab.io

Questions

• Any requests for specific visualizations?
• Any data that we're not tracking which should be tracked?

[achingbrain]

One thing here is that you are requesting older path gateway style URLs, e.g. https://ipfs.io/ipfs/QmcD6paKVWoN81QZZ7ftuxMura79GHHvWTxBT5fxWzufsk

Path gateways like ipfs.io are inherently insecure because all content accessed in this way in a browser shares an origin (e.g. host + port), which means websites accessed like this can view each others credentials which exposes users to phishing attacks.

Inbrowser.link only supports retrieving content using the more modern subdomain gateway style URLs, e.g. https://bafybeigocr7lc57bjw3b7jkv2y3mtggcvfiebs5wq5b7epfwtb5wgwgzm4.ipfs.inbrowser.link (this is the same CID as above in base32 CIDv1) - these do not share an origin and so are safer for users.

Requesting the path gateway style URL from inbrowser.link will first redirect you to the subdomain version before content is served so the ttfb measurement will always be impacted due to the extra redirects which keep our users safe.

Unfortunately people will use these stats to compare (for example) ipfs.io and inbrowser.link so it would be better to request the subdomain gateway style URL directly for a better comparison.

[dennis-tra]

I've reduced the list of monitored gateways to the following for now:

dweb.link
ipfs.io
ipfs.filebase.io
ipfs.orbitor.dev
w3s.link

Currently, we probe all gateways using the path-style resolution: /ipfs/{cid}. While this provides a consistent baseline, it creates a performance discrepancy for gateways like w3s.link that prioritize security by redirecting to subdomains.

I would argue that testing via the path-style is more representative of the average user’s initial interaction. However, I recognize the risk mentioned by @achingbrain: these metrics might inadvertently encourage users to prioritize speed over the security benefits of subdomain isolation. If subdomain support were universal across all gateways, I would favor using that as our primary testing method, but the current lack of parity makes path-based testing the most viable 'apples-to-apples' comparison in terms of resembling the end-user experience for now.

Regarding caching, my current grouping relies on edge headers (e.g., cf-cache-status, x-filebase-edge-cache, or x-cache). This misses deeper caching layers of the stack. Are there additional signals I should monitor to achieve more granular visibility into the cache status?

Btw, here's a preliminary graph: https://probelab.io/ipfs/gateways/#chart-ipfs-http-gateway-ttfb-ts (it only has data from two days so far, so doesn't look to exciting):

Next up, I'll add service worker gateway stats 👍

[dennis-tra]

Hi folks, I realize that it's tricky to accurately reflect the service worker time measurements. The download latency depends on the following:

• Which region was the request made from?
• Was it a controlled or an organic CID that was requested?
• Is this the total TTFB (including SW initialization) or the final TTFB (when everything's loaded and we are already on the subdomain)?
• Was the content retrieved directly from the peer or trustless gateway? (the former rarely happens, so we don’t have many datapoints)
• If the data was retrieved from the gateway: was it a cache hit or miss?

I guess I'd just add these ^^ parameters as select boxes in the graph. Otherwise we will always conflate data points that shouldn't be mixed.

For reference: I've just merged the PR: probe-lab/tiros#20