feat(s3-perf-test): Create s3-perf-test image #88
Conversation
|
@ADustyOldMuffin Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/13701773849 |
This will run a perf test on a given S3 bucket reading a set amount of objects in 16KiB blocks to simulate a real world workload on an object store.
ADustyOldMuffin
force-pushed
the
dhix/add-s3-perf-test
branch
from
41c4de8 to
ad7d764
Compare
|
@ADustyOldMuffin Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/13704759020 |
|
@ADustyOldMuffin Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/13709547872 |
ADustyOldMuffin
force-pushed
the
dhix/add-s3-perf-test
branch
from
73eb297 to
ec41135
Compare
|
@ADustyOldMuffin Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/13731442141 |
ADustyOldMuffin
force-pushed
the
dhix/add-s3-perf-test
branch
from
ec41135 to
90ad860
Compare
|
@ADustyOldMuffin Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/13731977537 |
ADustyOldMuffin
force-pushed
the
dhix/add-s3-perf-test
branch
from
90ad860 to
c99ba0c
Compare
|
@ADustyOldMuffin Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/13732136955 |
|
|
||
| "github.com/hashicorp/go-cleanhttp" | ||
| "github.com/jedib0t/go-pretty/v6/table" | ||
| "github.com/minio/minio-go/v7" |
There was a problem hiding this comment.
Far more convenient essentially! No code for endpoint resolvers, pointers, etc. Per is more or less the same since they both use the underlying net/http.
| for i := range threads { | ||
| wg.Add(1) | ||
| go func() { | ||
| result := runTest(ctx, params, i) |
There was a problem hiding this comment.
something that's nice is the ability for all the tests on different nodes to coordinate enough to gang-schedule themselves at approximately the exact same time in the cluster (esp. for various types of workloads)
There was a problem hiding this comment.
Ah sort of like a sync barrier almost?
There was a problem hiding this comment.
Ah sort of like a sync barrier almost?
yes, precisely that (we use NFS for coordination, since it needs to be low-dep / cluster agnostic ideally)
| continue | ||
| } | ||
|
|
||
| amount, err := io.Copy(io.Discard, resp) |
There was a problem hiding this comment.
if developing write workloads as well, I've found pooling/buffer reuse helpful
| s3Client, err := minio.New(endpoint, &minio.Options{ | ||
| Creds: credentials.NewStaticV4(accessKeyID, secretKey, ""), | ||
| Secure: ssl, | ||
| Transport: transport, |
There was a problem hiding this comment.
curious, why this http transport implementation?
There was a problem hiding this comment.
cleanhttp just sets sane defaults on the normal golang net/http. I then took some known good defaults for specifically load testing and then overwrite the defaults that the cleanhttp doesn't set/sets to something I don't want. Essentially its just a quicker short hand, but realistically not needed.
|
@ADustyOldMuffin Are we done with this? Merge? |
This will run a perf test on a given S3 bucket reading a set amount of objects in 16KiB blocks to simulate a real world workload on an object store.