feat: fio benchmark infrastructure mirroring mountpoint-s3#25
Draft
feat: fio benchmark infrastructure mirroring mountpoint-s3#25
Conversation
Benchmark Results (fio small, FUSE)
|
POSIX Compliance (pjdfstest) |
XciD
force-pushed
the
feat/fio-bench-infra
branch
from
f64a647 to
52ebe69
Compare
XciD
added a commit
that referenced
this pull request
Mar 9, 2026
## Summary - Add `--direct-io` CLI flag (default: off) that sets `FOPEN_DIRECT_IO` on file open/create - When enabled, every read goes through the FUSE handler, bypassing the kernel page cache - When enabled, the prefetch buffer becomes forward-only: consumed bytes are drained after serving, preventing re-reads from hitting the buffer (must refetch from CAS) - Without the flag (default), the kernel page cache handles cross-call caching and the prefetch buffer retains data for re-reads Useful for benchmarking real CAS throughput without page cache or buffer inflation. Not recommended for production (disables efficient mmap caching for safetensors workloads). The bench script (PR #25) will use `HF_DIRECT_IO=1` to enable this during fio runs.
XciD
force-pushed
the
feat/fio-bench-infra
branch
3 times, most recently
from
8a6ab43 to
e81e8fd
Compare
Add the same fio workloads and benchmark scripts as mountpoint-s3/scripts/, enabling direct throughput and latency comparisons. fio job files (scripts/fio/) are copied verbatim from mountpoint-s3: - read/: seq_read, rand_read, 4-thread, direct-IO, small-file variants - write/: seq_write, seq_write_direct - read_latency/: ttfb, ttfb_small Scripts: - scripts/fs_bench.sh: throughput benchmark (mirrors fs_bench.sh from mountpoint-s3). Creates a temp bucket, writes files write-through via hf-mount, then runs fio read and write jobs. Set HF_BENCH_BUCKET to reuse a pre-existing bucket. - scripts/fs_latency_bench.sh: TTFB latency benchmark (mirrors fs_latency_bench.sh). CI (.github/workflows/bench_perf.yml): - Triggers on push to main (publishes to gh-pages) and PRs labeled 'performance'. - Uses benchmark-action/github-action-benchmark for historical throughput and latency charts. - CI uses HF_JOB_NAME_FILTER=small and iterations=3 for fast runs; remove for full 100G comparison. doc/BENCHMARKING.md documents the workloads, how to run, and the comparison methodology.
Allows benchmarking without the on-disk xorb cache, comparable to mountpoint-s3 without --cache. Reads go directly to CAS on each FUSE miss (OS page cache still applies). The FileDownloadSession API already supported None for chunk_cache; this wires it up via a CLI flag and a HF_NO_DISK_CACHE=1 env var in the benchmark scripts.
Copy mix_1r4w, mix_2r2w, mix_4r1w from mountpoint-s3 and add run_benchmarks mix to fs_bench.sh.
Remove tests/bench.rs, tests/fio_bench.rs, and tests/common/bench.rs (superseded by scripts/fs_bench.sh fio benchmarks). Add scripts/posix_test.sh as a standalone pjdfstest runner with the same exclusion logic and regression baselines as tests/pjdfstest.rs.
Remove scripts/posix_test.sh (keeping cargo test). Rename test_pjdfstest to test_pjdfstest_fuse and add test_pjdfstest_nfs that runs the same POSIX conformance suite over NFS mount.
Add missing fio jobs from mountpoint-s3: create (100/1k/10k/100k files) and write_latency (1B file TTFB). Merge fs_latency_bench.sh into fs_bench.sh. Support FUSE/NFS backend, cache/no-cache, and advanced-writes via env vars (HF_MOUNT_BACKEND, HF_NO_DISK_CACHE, HF_ADVANCED_WRITES). Results are stored per mode in results/<mode>/.
- ci.yml bench job: use fs_bench.sh instead of removed cargo bench tests - bench_perf.yml: use unified script for both throughput and latency, fix results path (results/fuse/output.json)
Read benchmarks now write actual data through FUSE before measuring read throughput, instead of using fio --create_only=1 which only creates empty/sparse files. This ensures reads hit CAS and measure real download performance. - New run_read_category(): writes data in phase 1, reads on fresh mount in phase 2 - make_write_job(): generates a write job from a read job (strips time_based, direct, swaps rw) - read_latency also uses real data instead of create_only
Every read now goes through our FUSE handler instead of being served from the kernel page cache. This gives accurate benchmark numbers for CAS throughput and matches mountpoint-s3's behavior. Also fix fs_bench.sh: add --uid/--gid for sudo mounts, use create_on_open=1 for populate, and fail on mount errors.
…for prefilled buckets
XciD
force-pushed
the
feat/fio-bench-infra
branch
from
a199dd5 to
7b97a88
Compare
Throughput Benchmark Results
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/fs_bench.shscripts/fs_bench.shsupporting multiple mount modes via env vars:HF_MOUNT_BACKEND=fuse|nfsHF_NO_DISK_CACHE=1(comparable to mountpoint-s3 without--cache)HF_ADVANCED_WRITES=1HF_JOB_NAME_FILTER=smallfor quick CI runsHF_CATEGORIES=read,writeto run specific categoriesresults/<mode>/output.jsontests/bench.rs,tests/fio_bench.rs) superseded by fiotest_pjdfstest_nfs)doc/BENCHMARKING.md