Benchmark Infra Stability Assessment with private AWS devices

[guangy10]

Benchmark Infra Stability Assessment with private AWS devices

TL;DR
Here are the highlights and lowlights of the what plaint data shows.

• LLM models on private S22 (Android 13) demonstrate good stability with CV values of 2.3-2.6%
• The tiny non-genAI CPU-model (MobileNetV3+XNNPACK) shows sensitivity across all Android devices, though the rooted Pixel 3 provides relatively better stability (CV = 7.68%)
• QNN delegate (MobileNetV3+QNN) shows excellent stability on with CV values of 0.9-2.3%
• All iOS benchmarks exhibit poor stability regardless of device type (private/public), suggesting issues with our benchmarking methodology that need addressing before meaningful conclusions can be made

I am sharing my conclusions and recommendations based on the data at the end.

Understanding Stability Metrics

To properly assess the stability of ML model inference latency, I use several key statistical metrics:

• Coefficient of Variation (CV) / RSD
• Percentile-Based Metrics: P50/P90/P95/P99
• Interquartile Range (IQR)
• Max-Min Range Ratio
• Intra-Jitter Coefficient

And a composite stability score (0-100 scale) is calculated using weighted CV, Max/Min ratio, and P99/P50 ratio.

Intra-primary (private) Dataset Stability Comparison

Overall Summary:

Model	Device	Mean Latency (ms)	CV (%)	Stability Score	Stability Rating	Max/Min Ratio	P99/P50 Ratio
mv3_qnn	s22ultra_android14	1.01	0.91	93.81	Excellent	1.09	1.02
llama3_spinq	s22_android13	21771.59	2.36	84.70	Good	1.15	1.07
llama3_qlora	s22_android13	22502.10	2.64	83.37	Good	1.14	1.07
mv3_qnn	s22_android13	1.01	2.34	82.41	Good	1.19	1.14
llama3_qlora	s22ultra_android14	25022.84	6.18	62.54	Moderate	1.27	1.13
llama3_spinq	s22ultra_android14	24761.78	6.27	60.28	Moderate	1.36	1.15
mv3_xnnq8	pixel3_rooted_android	5.93	7.68	46.93	Poor	1.70	1.24
mv3_xnnq8	s22_android13	2.73	23.03	14.94	Poor	2.44	1.68
mv3_xnnq8	iphone15max_ios17	13.98	24.60	10.82	Poor	3.29	1.57
llama3_qlora	iphone15max_ios17	15239.42	29.97	6.24	Poor	2.50	2.36
mv3_xnnq8	s22ultra_android14	2.91	39.08	0.00	Poor	5.61	2.33
llama3_spinq	iphone15max_ios17	14440.01	36.79	0.00	Poor	3.62	2.94

Device-based Comparison:

Device	('Stability Score', 'mean')	('Stability Score', 'min')	('Stability Score', 'max')	('CV (%)', 'mean')	('CV (%)', 'min')	('CV (%)', 'max')
s22_android13	66.36	14.94	84.70	7.59	2.34	23.03
s22ultra_android14	54.16	0.00	93.81	13.11	0.91	39.08
pixel3_rooted_android	46.93	46.93	46.93	7.68	7.68	7.68
iphone15max_ios17	5.69	0.00	10.82	30.45	24.60	36.79

The analysis of latency stability across private AWS devices reveals certain patterns in performance consistency. Here are the insights and recommendations:

• s22_android13 provides the most stable environment for various model execution.
• mv3_qnn shows the most consistent performance across devices.
• mv3_xnnq8 shows more variability and may need further optimization. The rooted device still provides the best stability.
• iphone15max_ios17 shows higher variability in all setups and need further investigation.

Inter-dataset (private & public) Stability Comparison

NOTE: It's sad that the Dashboard does not provide all data needed for inter-dataset comparison. The reference (public) table is created based on whatever it's available.

Primary (Private) Datasets Summary:

Dataset	Model	Device	Mean Latency (ms)	CV (%)	Stability Score	Stability Rating
mv3_qnn+s22ultra_android14	mv3_qnn	s22ultra_android14	1.01	0.91	93.81	Excellent
llama3_spinq+s22_android13	llama3_spinq	s22_android13	21771.59	2.36	84.70	Good
llama3_qlora+s22_android13	llama3_qlora	s22_android13	22502.10	2.64	83.37	Good
mv3_qnn+s22_android13	mv3_qnn	s22_android13	1.01	2.34	82.41	Good
llama3_qlora+s22ultra_android14	llama3_qlora	s22ultra_android14	25022.84	6.18	62.54	Moderate
llama3_spinq+s22ultra_android14	llama3_spinq	s22ultra_android14	24761.78	6.27	60.28	Moderate
mv3_xnnq8+pixel3_rooted_android	mv3_xnnq8	pixel3_rooted_android	5.93	7.68	46.93	Poor
mv3_xnnq8+s22_android13	mv3_xnnq8	s22_android13	2.73	23.03	14.94	Poor
mv3_xnnq8+iphone15max_ios17	mv3_xnnq8	iphone15max_ios17	13.98	24.60	10.82	Poor
llama3_qlora+iphone15max_ios17	llama3_qlora	iphone15max_ios17	15239.42	29.97	6.24	Poor
mv3_xnnq8+s22ultra_android14	mv3_xnnq8	s22ultra_android14	2.91	39.08	0.00	Poor
llama3_spinq+iphone15max_ios17	llama3_spinq	iphone15max_ios17	14440.01	36.79	0.00	Poor

Reference (Public) Datasets Summary:

Dataset	Model	Device	Mean Latency (ms)	CV (%)	Stability Score	Stability Rating
mv3_qnn+s22ultra_android14	mv3_qnn	s22ultra_android14	1.02	1.35	90.39	Excellent
mv3_xnnq8+s22ultra_android14	mv3_xnnq8	s22ultra_android14	3.63	22.35	15.48	Poor
llama3_qlora+iphone15max_ios17	llama3_qlora	iphone15max_ios17	14133.01	21.37	10.57	Poor
llama3_spinq+iphone15max_ios17	llama3_spinq	iphone15max_ios17	13118.40	21.76	2.65	Poor
mv3_xnnq8+s22_android13	mv3_xnnq8	s22_android13	1.92	55.09	0.00	Poor
mv3_qnn+s22_android13	mv3_qnn	s22_android13	1.44	57.29	0.00	Poor
mv3_spinq+iphone15max_ios17	mv3_spinq	iphone15max_ios17	5.05	136.34	0.00	Poor

Private vs Public Comparison:

Dataset	Model	Device	Private Score	Public Score	Score Diff	Private CV (%)	Public CV (%)	CV Diff (%)
mv3_qnn+s22_android13	mv3_qnn	s22_android13	82.41	0.00	82.41	2.34	57.29	-54.95
mv3_xnnq8+s22_android13	mv3_xnnq8	s22_android13	14.94	0.00	14.94	23.03	55.09	-32.06
mv3_qnn+s22ultra_android14	mv3_qnn	s22ultra_android14	93.81	90.39	3.42	0.91	1.35	-0.44
llama3_spinq+iphone15max_ios17	llama3_spinq	iphone15max_ios17	0.00	2.65	-2.65	36.79	21.76	15.04
llama3_qlora+iphone15max_ios17	llama3_qlora	iphone15max_ios17	6.24	10.57	-4.33	29.97	21.37	8.60
mv3_xnnq8+s22ultra_android14	mv3_xnnq8	s22ultra_android14	0.00	15.48	-15.48	39.08	22.35	16.73

Overall Insights and Recommendations

• Overall private Android devices provides better stability over public devices. With rooted pixel + s22, we can get good stability coverage.
• High variability in ALL iOS dataset regardless it's private or public, indicating issue of app/benchmark config, e.g. no warmup, no cooldown, too less iters, etc. Further investigation is needed.

Detailed Stability Analysis on Individual Dataset - Primary (Private)

The full list of individual dataset analysis can be downloaded here. In this section I will highlight detailed statistical metrics for only a few selected datasets.

1. Latency Stability Analysis: llama3_spinq+s22_android13 (Primary)

================================================================================
Model: llama3_spinqant
Device: s22_android13

Dataset Overview:
  - Number of samples: 88
  - Date range: 2025-04-29 09:48:57+00:00 to 2025-05-13 21:08:36+00:00

Central Tendency Metrics:
  - Mean latency: 21771.59 ms
  - Median latency (P50): 21668.24 ms
  - Mean trimmed latency: 21662.53 ms
  - Median trimmed latency: 21559.89 ms

Dispersion Metrics:
  - Standard deviation: 514.89 ms
  - Coefficient of variation (CV): 2.36%
  - Interquartile range (IQR): 602.75 ms
  - Trimmed standard deviation: 515.03 ms
  - Trimmed coefficient of variation: 2.38%

Percentile Metrics:
  - P50 (median): 21668.24 ms
  - P90: 22438.74 ms
  - P95: 22542.42 ms
  - P99: 23104.76 ms

Inter-Jitter Metrics (variability between runs):
  - Max/Min ratio: 1.1452
  - P99/P50 ratio: 1.0663
  - Mean rolling std (window=5): 449.10 ms

Intra-Jitter Metrics (variability within runs):
  - Mean trimming effect ratio: 0.50%
  - Max trimming effect ratio: 0.89%

Throughput Metrics:
  - Mean TPS: 33.76
  - TPS coefficient of variation: 4.70%

Stability Assessment:
  - Overall stability score: 84.7/100
  - Overall stability rating: Good

Interpretation:
  The benchmark shows good stability (score: 84.7/100) with low
  variation between runs (CV: 2.36%).
  Performance is consistent and predictable for most use cases.

================================================================================

2. Latency Stability Analysis: mv3_qnn+s22_android13 (Primary)

================================================================================
Model: mv3_qnn
Device: s22_android13

Dataset Overview:
  - Number of samples: 100
  - Date range: 2025-04-29 09:48:57+00:00 to 2025-05-15 21:14:41+00:00

Central Tendency Metrics:
  - Mean latency: 1.01 ms
  - Median latency (P50): 1.00 ms
  - Mean trimmed latency: 1.00 ms
  - Median trimmed latency: 1.00 ms

Dispersion Metrics:
  - Standard deviation: 0.02 ms
  - Coefficient of variation (CV): 2.34%
  - Interquartile range (IQR): 0.01 ms
  - Trimmed standard deviation: 0.02 ms
  - Trimmed coefficient of variation: 2.27%

Percentile Metrics:
  - P50 (median): 1.00 ms
  - P90: 1.01 ms
  - P95: 1.01 ms
  - P99: 1.14 ms

Inter-Jitter Metrics (variability between runs):
  - Max/Min ratio: 1.1919
  - P99/P50 ratio: 1.1404
  - Mean rolling std (window=5): 0.01 ms

Intra-Jitter Metrics (variability within runs):
  - Mean trimming effect ratio: 0.19%
  - Max trimming effect ratio: 1.00%

Stability Assessment:
  - Overall stability score: 82.4/100
  - Overall stability rating: Good

Interpretation:
  The benchmark shows good stability (score: 82.4/100) with low
  variation between runs (CV: 2.34%).
  Performance is consistent and predictable for most use cases.

  The P99/P50 ratio of 1.14 suggests
  occasional latency spikes that could affect tail latency sensitive applications.

================================================================================

3. Latency Stability Analysis: mv3_xnnq8+s22_android13 (Primary)

================================================================================
Model: mv3_xnnq8
Device: s22_android13

Dataset Overview:
  - Number of samples: 88
  - Date range: 2025-04-29 09:48:57+00:00 to 2025-05-13 21:08:36+00:00

Central Tendency Metrics:
  - Mean latency: 2.73 ms
  - Median latency (P50): 2.65 ms
  - Mean trimmed latency: 2.22 ms
  - Median trimmed latency: 2.10 ms

Dispersion Metrics:
  - Standard deviation: 0.63 ms
  - Coefficient of variation (CV): 23.03%
  - Interquartile range (IQR): 0.95 ms
  - Trimmed standard deviation: 0.36 ms
  - Trimmed coefficient of variation: 15.98%

Percentile Metrics:
  - P50 (median): 2.65 ms
  - P90: 3.59 ms
  - P95: 3.74 ms
  - P99: 4.46 ms

Inter-Jitter Metrics (variability between runs):
  - Max/Min ratio: 2.4427
  - P99/P50 ratio: 1.6812
  - Mean rolling std (window=5): 0.60 ms

Intra-Jitter Metrics (variability within runs):
  - Mean trimming effect ratio: 16.52%
  - Max trimming effect ratio: 36.96%

Stability Assessment:
  - Overall stability score: 14.9/100
  - Overall stability rating: Poor

================================================================================

4. Latency Stability Analysis: llama3_spinq+iphone15max_ios17 (Primary)

================================================================================
Model: llama3_spinquant
Device: iphone15max_ios17

Dataset Overview:
  - Number of samples: 72
  - Date range: 2025-04-29 21:26:38+00:00 to 2025-05-10 09:24:40+00:00

Central Tendency Metrics:
  - Mean latency: 14440.01 ms
  - Median latency (P50): 12149.50 ms

Dispersion Metrics:
  - Standard deviation: 5312.72 ms
  - Coefficient of variation (CV): 36.79%
  - Interquartile range (IQR): 2231.00 ms

Percentile Metrics:
  - P50 (median): 12149.50 ms
  - P90: 18765.00 ms
  - P95: 25178.50 ms
  - P99: 35673.00 ms

Inter-Jitter Metrics (variability between runs):
  - Max/Min ratio: 3.6163
  - P99/P50 ratio: 2.9362
  - Mean rolling std (window=5): 3238.06 ms

Throughput Metrics:
  - Mean TPS: 11.66
  - TPS coefficient of variation: 38.53%

Stability Assessment:
  - Overall stability score: 0.0/100
  - Overall stability rating: Poor

================================================================================

Conclusion and Recommendations for Next Steps

Android Benchmarking

The analysis shows that private AWS devices provide significantly better stability for Android benchmarking. The data supports a specific configuration strategy:

• S22 with Android 13 provides excellent stability for LLM models (CV values of 2.3-2.6%) and QNN delegate (CV of 2.34%)
• Rooted Pixel 3 offers the best stability for tiny non-genAI models (CV of 7.68% for MV3 XNNQ8)
  As next steps, we should:

1. Expand the size of the private pool and start running more models+configs. By end of June, I hope we will have sufficient data to make a definitive go/no-go decision.
2. At meanwhile, investigate MV3 XNNQ8 variability. If we can root cause and stabilize it, we won't need to run anything on rooted device.

iOS Benchmarking

Both private and public iOS devices show poor stability across all models (CV values of 21-37%), indicating fundamental flaws in our iOS benchmarking methodology/app. Until methodology improvements have been validated with new benchmark data, we're at the risk of having a meaningful conclusion of moving iOS benchmarking forward by end of June.

DevX Improvements

Our current benchmarking infrastructure has critical gaps that limit our ability to understand and address stability issues. These limitations are particularly problematic when trying to diagnose the root causes of performance variations we've observed across devices.

Current Gaps

1. Lack of Device Telemetry: We currently have no mechanism to collect essential device state information during benchmarking runs, including:
   • CPU frequency and thermal state data
   • Memory usage patterns and garbage collection events
   • Background process activity that may interfere with benchmarks
   • Battery state and power mode settings that can affect performance
2. Limited Analysis Capabilities and Manual Process: Our infrastructure lacks built-in stability assessment tools, forcing a time-consuming manual process:
   • No automated calculation of key statistical metrics (CV, IQR, P99/P50), and no standardized criteria for interpreting results
   • Manual data extraction from the benchmark dashboard
   • No way to correlate performance variations with individual device and its state

Addressing these gaps is urgent to establish a reliable benchmarking infrastructure. Without these improvements, we risk making timely decisions and basing conclusions on misleading or incomplete data.

References

Here I attached the source of data and my script if anyone want to repeat the work. Please also use it as a reference when filling the infra gaps above.

The script used for analysis

• PR: [Not To Land] Script for benchmark satbility assessment #10982

Data source:

• Datasets from Primary/Private AWS devices:
  Benchmark Dataset with Private AWS Devices.xlsx

• Datasets from Reference/Public AWS devices:
  Benchmark Dataset with Public AWS Devices.xlsx

Each tab represent one dataset collected with one model+config+device combination. The source of the data are copied from the ExecuTorch benchmark dashboard

[kimishpatel]

THis is a good analysis. Let me dive into it a bit

[GregoryComer]

Thanks for compiling this. Looking at the next steps, are we intending to analyze the impact of warmup iterations or high iteration counts? It would be very interesting to see the data on runtimes per iteration over say 1000 iterations run back to back. Does it stabilize? Or randomly spike? Can we rely on median latency being reliable over a large number of iterations?