What to do with performance outliers

[ryanblock]

Lately we've observed some extremely long latencies in IAM requests; I'd expect 200-500ms, but in some cases I've seen >15 second response times.

I added some additional logging for such scenarios to our benchmarking tools; here are some examples from yesterday's run:

...
[Benchmark] Detected unusually slow operation at 2024-08-05T01:50:25.358Z:
aws-sdk-v2-raw > f32e644b-c205-4564-9ac5-bb3cfd528415 > readIAM
{
  memoryStart: 67117056,
  memoryEnd: 67076096,
  timeStart: 1722822625358,
  timeEnd: 1722822641486,
  time: 16128,
  memory: -40960
}
...
[Benchmark] Detected unusually slow operation at 2024-08-05T01:51:07.041Z:
aws-sdk-v2-bundled > c21496ac-955b-42d8-983c-81fb0d9bf8bb > readIAM
{
  memoryStart: 79491072,
  memoryEnd: 75857920,
  timeStart: 1722822667041,
  timeEnd: 1722822674555,
  time: 7514,
  memory: -3633152
}
[Benchmark] Detected unusually slow operation at 2024-08-05T01:52:19.166Z:
aws-sdk-v2-bundled > 35534741-d829-4be0-8190-89bd0e1565bc > readIAM
{
  memoryStart: 78278656,
  memoryEnd: 75649024,
  timeStart: 1722822739166,
  timeEnd: 1722822755154,
  time: 15988,
  memory: -2629632
}
...
[Benchmark] Detected unusually slow operation at 2024-08-05T01:53:28.520Z:
aws-sdk-v2-bundled > 25e2e394-28b0-4dd0-8a7d-33d0bdfac1da > writeIAM
{
  memoryStart: 78168064,
  memoryEnd: 74248192,
  timeStart: 1722822808520,
  timeEnd: 1722822824197,
  time: 15677,
  memory: -3919872
}

I'm considering options to remediate. On the one hand, the response times are the response times, so shouldn't that be what goes into the results?

On the other hand, the aws-lite performance project is attempting to benchmark clients, not APIs, so through that lens perhaps it makes sense to throw out outlier responses of egregious latency?

Feedback welcome!

[brianleroux]

🤔 thinking we need to discard, but somehow flag/raise these outliers so we can get in touch w folks at AWS about it…seems like a problem they'd want to address esp from an SLA perspective