[Feature] Tool services

[vmoens]

Stack from ghstack (oldest at bottom):

• -> [Feature] Tool services #3220
• [Feature] Aggregation strategies #3209
• [Feature] kl_mask_threshold #3208
• [Feature] CISPO #3207

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3220

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 10 New Failures, 3 Unrelated Failures

As of commit fd322d9 with merge base 13434eb ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 139.
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t e091e4f881ab1fc5b1e4a833f6eadcb3851e6f39c93ce3f35be9d3bef1f24dbe /exec failed with exit code 1
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
  RuntimeError: SumSegmentTreeFp32 is not available. See warning above.
• Push Binary Nightly / build-wheel-unix (linux, ubuntu-22.04, 3.9, cp39-cp39, cpu, cpu) (gh)
• Push Binary Nightly / build-wheel-unix (macos, macos-latest, 3.9, cp39-cp39, cpu, cpu) (gh)
  Process completed with exit code 1.
• Push Binary Nightly / build-wheel-windows (3.9, 3.9) (gh)
• Push Binary Nightly / test-wheel-unix (linux, ubuntu-22.04, 3.9, cp39-cp39, cpu, cpu) (gh)
  Process completed with exit code 1.
• Push Binary Nightly / test-wheel-unix (macos, macos-latest, 3.9, cp39-cp39, cpu, cpu) (gh)
  Process completed with exit code 1.
• Push Binary Nightly / test-wheel-windows (3.9, 3.9) (gh)
  Process completed with exit code 1.
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t ae466f479a49476438d667234c3ee6f6cec621b9eea0ad0871077b4d4c21c1cc /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh) (trunk failure)
  test/test_collector.py::TestPolicyFactory::test_weight_update[scheme_pipe]
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh) (trunk failure)
  test/test_collector.py::TestPolicyFactory::test_weight_update[scheme_pipe]
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh) (trunk failure)
  test/test_collector.py::TestPolicyFactory::test_weight_update[scheme_pipe]

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 154. Improved: $\large\color{#35bf28}26$. Worsened: $\large\color{#d91a1a}8$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_tensor_to_bytestream_speed[pickle]	83.3977μs	82.2289μs	12.1612 KOps/s	12.2270 KOps/s	$\color{#d91a1a}-0.54\%$
test_tensor_to_bytestream_speed[torch.save]	0.1417ms	0.1402ms	7.1332 KOps/s	7.0779 KOps/s	$\color{#35bf28}+0.78\%$
test_tensor_to_bytestream_speed[untyped_storage]	0.1262s	0.1258s	7.9494 Ops/s	7.1918 Ops/s	$\textbf{\color{#35bf28}+10.53\%}$
test_tensor_to_bytestream_speed[numpy]	2.8534μs	2.8464μs	351.3181 KOps/s	346.1419 KOps/s	$\color{#35bf28}+1.50\%$
test_tensor_to_bytestream_speed[safetensors]	44.8393μs	44.0607μs	22.6960 KOps/s	23.4494 KOps/s	$\color{#d91a1a}-3.21\%$
test_simple	0.5511s	0.5508s	1.8155 Ops/s	1.7256 Ops/s	$\textbf{\color{#35bf28}+5.21\%}$
test_transformed	1.2291s	1.1349s	0.8812 Ops/s	0.8778 Ops/s	$\color{#35bf28}+0.38\%$
test_serial	1.6745s	1.6707s	0.5985 Ops/s	0.5875 Ops/s	$\color{#35bf28}+1.88\%$
test_parallel	1.1808s	1.1013s	0.9080 Ops/s	0.9015 Ops/s	$\color{#35bf28}+0.72\%$
test_step_mdp_speed[True-True-True-True-True]	0.2565ms	45.2691μs	22.0901 KOps/s	22.5514 KOps/s	$\color{#d91a1a}-2.05\%$
test_step_mdp_speed[True-True-True-True-False]	50.0410μs	25.6700μs	38.9560 KOps/s	39.4071 KOps/s	$\color{#d91a1a}-1.14\%$
test_step_mdp_speed[True-True-True-False-True]	59.7510μs	25.2690μs	39.5742 KOps/s	39.8418 KOps/s	$\color{#d91a1a}-0.67\%$
test_step_mdp_speed[True-True-True-False-False]	44.9300μs	14.0725μs	71.0608 KOps/s	74.3307 KOps/s	$\color{#d91a1a}-4.40\%$
test_step_mdp_speed[True-True-False-True-True]	86.8620μs	48.9648μs	20.4228 KOps/s	21.1647 KOps/s	$\color{#d91a1a}-3.51\%$
test_step_mdp_speed[True-True-False-True-False]	61.1010μs	28.3282μs	35.3005 KOps/s	36.2483 KOps/s	$\color{#d91a1a}-2.61\%$
test_step_mdp_speed[True-True-False-False-True]	61.3110μs	28.6693μs	34.8806 KOps/s	35.8868 KOps/s	$\color{#d91a1a}-2.80\%$
test_step_mdp_speed[True-True-False-False-False]	45.4410μs	16.8164μs	59.4658 KOps/s	59.9908 KOps/s	$\color{#d91a1a}-0.88\%$
test_step_mdp_speed[True-False-True-True-True]	85.9010μs	50.9522μs	19.6262 KOps/s	19.7770 KOps/s	$\color{#d91a1a}-0.76\%$
test_step_mdp_speed[True-False-True-True-False]	55.9310μs	30.7293μs	32.5422 KOps/s	32.5933 KOps/s	$\color{#d91a1a}-0.16\%$
test_step_mdp_speed[True-False-True-False-True]	54.7910μs	28.4486μs	35.1511 KOps/s	35.3310 KOps/s	$\color{#d91a1a}-0.51\%$
test_step_mdp_speed[True-False-True-False-False]	57.2810μs	16.9382μs	59.0381 KOps/s	59.9868 KOps/s	$\color{#d91a1a}-1.58\%$
test_step_mdp_speed[True-False-False-True-True]	86.9510μs	53.8120μs	18.5832 KOps/s	19.1720 KOps/s	$\color{#d91a1a}-3.07\%$
test_step_mdp_speed[True-False-False-True-False]	67.5910μs	33.6022μs	29.7599 KOps/s	30.2614 KOps/s	$\color{#d91a1a}-1.66\%$
test_step_mdp_speed[True-False-False-False-True]	61.1510μs	30.7317μs	32.5397 KOps/s	32.7764 KOps/s	$\color{#d91a1a}-0.72\%$
test_step_mdp_speed[True-False-False-False-False]	46.1210μs	19.5635μs	51.1156 KOps/s	51.7166 KOps/s	$\color{#d91a1a}-1.16\%$
test_step_mdp_speed[False-True-True-True-True]	88.8210μs	51.3562μs	19.4718 KOps/s	19.7918 KOps/s	$\color{#d91a1a}-1.62\%$
test_step_mdp_speed[False-True-True-True-False]	89.6710μs	30.5685μs	32.7134 KOps/s	32.4224 KOps/s	$\color{#35bf28}+0.90\%$
test_step_mdp_speed[False-True-True-False-True]	2.3074ms	32.4685μs	30.7991 KOps/s	31.4279 KOps/s	$\color{#d91a1a}-2.00\%$
test_step_mdp_speed[False-True-True-False-False]	47.8010μs	18.7374μs	53.3692 KOps/s	54.3079 KOps/s	$\color{#d91a1a}-1.73\%$
test_step_mdp_speed[False-True-False-True-True]	95.5510μs	54.0221μs	18.5110 KOps/s	18.8647 KOps/s	$\color{#d91a1a}-1.88\%$
test_step_mdp_speed[False-True-False-True-False]	58.5210μs	34.0829μs	29.3403 KOps/s	29.5556 KOps/s	$\color{#d91a1a}-0.73\%$
test_step_mdp_speed[False-True-False-False-True]	65.9510μs	35.0735μs	28.5115 KOps/s	29.0936 KOps/s	$\color{#d91a1a}-2.00\%$
test_step_mdp_speed[False-True-False-False-False]	84.9910μs	21.5254μs	46.4567 KOps/s	46.8121 KOps/s	$\color{#d91a1a}-0.76\%$
test_step_mdp_speed[False-False-True-True-True]	90.0610μs	56.6321μs	17.6578 KOps/s	17.7992 KOps/s	$\color{#d91a1a}-0.79\%$
test_step_mdp_speed[False-False-True-True-False]	62.1810μs	36.6843μs	27.2597 KOps/s	27.5638 KOps/s	$\color{#d91a1a}-1.10\%$
test_step_mdp_speed[False-False-True-False-True]	68.2710μs	34.7539μs	28.7738 KOps/s	28.8145 KOps/s	$\color{#d91a1a}-0.14\%$
test_step_mdp_speed[False-False-True-False-False]	48.9110μs	21.3743μs	46.7852 KOps/s	47.5748 KOps/s	$\color{#d91a1a}-1.66\%$
test_step_mdp_speed[False-False-False-True-True]	0.1236ms	58.5251μs	17.0867 KOps/s	17.6138 KOps/s	$\color{#d91a1a}-2.99\%$
test_step_mdp_speed[False-False-False-True-False]	79.7920μs	39.0344μs	25.6185 KOps/s	26.0650 KOps/s	$\color{#d91a1a}-1.71\%$
test_step_mdp_speed[False-False-False-False-True]	85.9210μs	36.6855μs	27.2587 KOps/s	27.2001 KOps/s	$\color{#35bf28}+0.22\%$
test_step_mdp_speed[False-False-False-False-False]	77.6510μs	23.6834μs	42.2237 KOps/s	41.7909 KOps/s	$\color{#35bf28}+1.04\%$
test_values[generalized_advantage_estimate-True-True]	10.3466ms	10.1817ms	98.2155 Ops/s	92.8830 Ops/s	$\textbf{\color{#35bf28}+5.74\%}$
test_values[vec_generalized_advantage_estimate-True-True]	19.5626ms	17.7422ms	56.3628 Ops/s	55.7797 Ops/s	$\color{#35bf28}+1.05\%$
test_values[td0_return_estimate-False-False]	0.2213ms	0.1300ms	7.6926 KOps/s	7.7042 KOps/s	$\color{#d91a1a}-0.15\%$
test_values[td1_return_estimate-False-False]	29.1941ms	28.1269ms	35.5532 Ops/s	34.5307 Ops/s	$\color{#35bf28}+2.96\%$
test_values[vec_td1_return_estimate-False-False]	18.1154ms	17.7895ms	56.2128 Ops/s	55.1529 Ops/s	$\color{#35bf28}+1.92\%$
test_values[td_lambda_return_estimate-True-False]	44.6164ms	43.0117ms	23.2495 Ops/s	22.0789 Ops/s	$\textbf{\color{#35bf28}+5.30\%}$
test_values[vec_td_lambda_return_estimate-True-False]	18.0226ms	17.7929ms	56.2022 Ops/s	55.8778 Ops/s	$\color{#35bf28}+0.58\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	9.0348ms	8.8796ms	112.6172 Ops/s	107.7865 Ops/s	$\color{#35bf28}+4.48\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	1.7368ms	1.5210ms	657.4554 Ops/s	664.7832 Ops/s	$\color{#d91a1a}-1.10\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.5451ms	0.4210ms	2.3755 KOps/s	2.2754 KOps/s	$\color{#35bf28}+4.40\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	35.1142ms	30.8360ms	32.4296 Ops/s	28.5483 Ops/s	$\textbf{\color{#35bf28}+13.60\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	2.2579ms	1.7661ms	566.2236 Ops/s	566.3072 Ops/s	$\color{#d91a1a}-0.01\%$
test_dqn_speed[False-None]	1.5954ms	1.4277ms	700.4305 Ops/s	683.5234 Ops/s	$\color{#35bf28}+2.47\%$
test_dqn_speed[False-backward]	2.0260ms	1.9552ms	511.4674 Ops/s	510.2858 Ops/s	$\color{#35bf28}+0.23\%$
test_dqn_speed[True-None]	0.9761ms	0.5264ms	1.8997 KOps/s	1.8919 KOps/s	$\color{#35bf28}+0.41\%$
test_dqn_speed[True-backward]	1.2817ms	0.9792ms	1.0213 KOps/s	849.0331 Ops/s	$\textbf{\color{#35bf28}+20.29\%}$
test_dqn_speed[reduce-overhead-None]	0.6536ms	0.5178ms	1.9313 KOps/s	1.8662 KOps/s	$\color{#35bf28}+3.49\%$
test_dqn_speed[reduce-overhead-backward]	1.0200ms	0.9626ms	1.0389 KOps/s	990.1317 Ops/s	$\color{#35bf28}+4.93\%$
test_ddpg_speed[False-None]	3.2171ms	2.9445ms	339.6142 Ops/s	342.4657 Ops/s	$\color{#d91a1a}-0.83\%$
test_ddpg_speed[False-backward]	4.3257ms	4.1793ms	239.2736 Ops/s	241.1040 Ops/s	$\color{#d91a1a}-0.76\%$
test_ddpg_speed[True-None]	1.8099ms	1.3981ms	715.2619 Ops/s	684.9999 Ops/s	$\color{#35bf28}+4.42\%$
test_ddpg_speed[True-backward]	2.4878ms	2.4073ms	415.4104 Ops/s	343.3575 Ops/s	$\textbf{\color{#35bf28}+20.98\%}$
test_ddpg_speed[reduce-overhead-None]	1.7608ms	1.3813ms	723.9520 Ops/s	713.1165 Ops/s	$\color{#35bf28}+1.52\%$
test_ddpg_speed[reduce-overhead-backward]	2.5060ms	2.4045ms	415.8848 Ops/s	415.9487 Ops/s	$\color{#d91a1a}-0.02\%$
test_sac_speed[False-None]	8.3992ms	7.9807ms	125.3028 Ops/s	125.1093 Ops/s	$\color{#35bf28}+0.15\%$
test_sac_speed[False-backward]	11.9397ms	11.2906ms	88.5693 Ops/s	88.6747 Ops/s	$\color{#d91a1a}-0.12\%$
test_sac_speed[True-None]	2.2830ms	2.0995ms	476.3009 Ops/s	447.2465 Ops/s	$\textbf{\color{#35bf28}+6.50\%}$
test_sac_speed[True-backward]	4.1260ms	4.0245ms	248.4781 Ops/s	238.1254 Ops/s	$\color{#35bf28}+4.35\%$
test_sac_speed[reduce-overhead-None]	2.2710ms	2.0887ms	478.7683 Ops/s	461.0710 Ops/s	$\color{#35bf28}+3.84\%$
test_sac_speed[reduce-overhead-backward]	4.3802ms	4.0468ms	247.1094 Ops/s	232.7222 Ops/s	$\textbf{\color{#35bf28}+6.18\%}$
test_redq_speed[False-None]	14.8827ms	10.5865ms	94.4596 Ops/s	91.8273 Ops/s	$\color{#35bf28}+2.87\%$
test_redq_speed[False-backward]	22.8870ms	18.3763ms	54.4179 Ops/s	52.9596 Ops/s	$\color{#35bf28}+2.75\%$
test_redq_speed[True-None]	4.9179ms	4.4771ms	223.3611 Ops/s	214.3383 Ops/s	$\color{#35bf28}+4.21\%$
test_redq_speed[True-backward]	10.3125ms	9.9981ms	100.0191 Ops/s	101.2214 Ops/s	$\color{#d91a1a}-1.19\%$
test_redq_speed[reduce-overhead-None]	4.5877ms	4.4067ms	226.9260 Ops/s	227.3249 Ops/s	$\color{#d91a1a}-0.18\%$
test_redq_speed[reduce-overhead-backward]	10.4699ms	10.1017ms	98.9934 Ops/s	99.3655 Ops/s	$\color{#d91a1a}-0.37\%$
test_redq_deprec_speed[False-None]	11.6452ms	11.1612ms	89.5963 Ops/s	90.9141 Ops/s	$\color{#d91a1a}-1.45\%$
test_redq_deprec_speed[False-backward]	16.6697ms	16.1025ms	62.1021 Ops/s	62.8677 Ops/s	$\color{#d91a1a}-1.22\%$
test_redq_deprec_speed[True-None]	4.0410ms	3.6399ms	274.7327 Ops/s	278.9679 Ops/s	$\color{#d91a1a}-1.52\%$
test_redq_deprec_speed[True-backward]	8.1393ms	7.7149ms	129.6195 Ops/s	133.5010 Ops/s	$\color{#d91a1a}-2.91\%$
test_redq_deprec_speed[reduce-overhead-None]	3.7786ms	3.6370ms	274.9503 Ops/s	278.3341 Ops/s	$\color{#d91a1a}-1.22\%$
test_redq_deprec_speed[reduce-overhead-backward]	7.9763ms	7.7180ms	129.5674 Ops/s	117.7162 Ops/s	$\textbf{\color{#35bf28}+10.07\%}$
test_td3_speed[False-None]	8.4542ms	8.0367ms	124.4299 Ops/s	121.6438 Ops/s	$\color{#35bf28}+2.29\%$
test_td3_speed[False-backward]	11.3941ms	11.0200ms	90.7440 Ops/s	89.2451 Ops/s	$\color{#35bf28}+1.68\%$
test_td3_speed[True-None]	1.8339ms	1.7897ms	558.7561 Ops/s	547.7652 Ops/s	$\color{#35bf28}+2.01\%$
test_td3_speed[True-backward]	3.7780ms	3.6689ms	272.5648 Ops/s	271.6451 Ops/s	$\color{#35bf28}+0.34\%$
test_td3_speed[reduce-overhead-None]	1.8395ms	1.7797ms	561.8776 Ops/s	559.6150 Ops/s	$\color{#35bf28}+0.40\%$
test_td3_speed[reduce-overhead-backward]	3.9713ms	3.8066ms	262.7027 Ops/s	271.0359 Ops/s	$\color{#d91a1a}-3.07\%$
test_cql_speed[False-None]	29.7301ms	26.9364ms	37.1245 Ops/s	38.0542 Ops/s	$\color{#d91a1a}-2.44\%$
test_cql_speed[False-backward]	39.9271ms	36.6344ms	27.2968 Ops/s	27.2306 Ops/s	$\color{#35bf28}+0.24\%$
test_cql_speed[True-None]	12.9378ms	12.4103ms	80.5779 Ops/s	75.6413 Ops/s	$\textbf{\color{#35bf28}+6.53\%}$
test_cql_speed[True-backward]	19.0922ms	18.7147ms	53.4340 Ops/s	53.8889 Ops/s	$\color{#d91a1a}-0.84\%$
test_cql_speed[reduce-overhead-None]	13.0720ms	12.5503ms	79.6795 Ops/s	78.0202 Ops/s	$\color{#35bf28}+2.13\%$
test_cql_speed[reduce-overhead-backward]	18.4075ms	17.7883ms	56.2169 Ops/s	55.7733 Ops/s	$\color{#35bf28}+0.80\%$
test_a2c_speed[False-None]	5.8077ms	5.4664ms	182.9366 Ops/s	183.3547 Ops/s	$\color{#d91a1a}-0.23\%$
test_a2c_speed[False-backward]	12.3888ms	12.0591ms	82.9246 Ops/s	83.4493 Ops/s	$\color{#d91a1a}-0.63\%$
test_a2c_speed[True-None]	3.8152ms	3.6893ms	271.0564 Ops/s	268.4752 Ops/s	$\color{#35bf28}+0.96\%$
test_a2c_speed[True-backward]	8.9198ms	8.7095ms	114.8173 Ops/s	95.4650 Ops/s	$\textbf{\color{#35bf28}+20.27\%}$
test_a2c_speed[reduce-overhead-None]	4.1133ms	3.7229ms	268.6053 Ops/s	265.9019 Ops/s	$\color{#35bf28}+1.02\%$
test_a2c_speed[reduce-overhead-backward]	9.1598ms	8.9286ms	112.0001 Ops/s	104.7267 Ops/s	$\textbf{\color{#35bf28}+6.95\%}$
test_ppo_speed[False-None]	6.0602ms	5.8654ms	170.4909 Ops/s	170.8173 Ops/s	$\color{#d91a1a}-0.19\%$
test_ppo_speed[False-backward]	13.0954ms	12.7779ms	78.2601 Ops/s	79.0036 Ops/s	$\color{#d91a1a}-0.94\%$
test_ppo_speed[True-None]	3.7781ms	3.6418ms	274.5916 Ops/s	264.0889 Ops/s	$\color{#35bf28}+3.98\%$
test_ppo_speed[True-backward]	8.7665ms	8.4886ms	117.8051 Ops/s	114.1525 Ops/s	$\color{#35bf28}+3.20\%$
test_ppo_speed[reduce-overhead-None]	3.7752ms	3.6425ms	274.5383 Ops/s	272.8544 Ops/s	$\color{#35bf28}+0.62\%$
test_ppo_speed[reduce-overhead-backward]	9.1717ms	8.7972ms	113.6729 Ops/s	105.1341 Ops/s	$\textbf{\color{#35bf28}+8.12\%}$
test_reinforce_speed[False-None]	4.8145ms	4.6239ms	216.2676 Ops/s	217.1706 Ops/s	$\color{#d91a1a}-0.42\%$
test_reinforce_speed[False-backward]	7.7059ms	7.4952ms	133.4191 Ops/s	132.8545 Ops/s	$\color{#35bf28}+0.43\%$
test_reinforce_speed[True-None]	3.7558ms	2.9478ms	339.2364 Ops/s	332.2490 Ops/s	$\color{#35bf28}+2.10\%$
test_reinforce_speed[True-backward]	8.0322ms	7.7510ms	129.0155 Ops/s	116.4172 Ops/s	$\textbf{\color{#35bf28}+10.82\%}$
test_reinforce_speed[reduce-overhead-None]	3.2986ms	2.8567ms	350.0517 Ops/s	327.5972 Ops/s	$\textbf{\color{#35bf28}+6.85\%}$
test_reinforce_speed[reduce-overhead-backward]	8.4663ms	8.0341ms	124.4702 Ops/s	123.2936 Ops/s	$\color{#35bf28}+0.95\%$
test_iql_speed[False-None]	25.7286ms	20.5354ms	48.6965 Ops/s	48.2719 Ops/s	$\color{#35bf28}+0.88\%$
test_iql_speed[False-backward]	35.8644ms	30.9496ms	32.3106 Ops/s	32.3455 Ops/s	$\color{#d91a1a}-0.11\%$
test_iql_speed[True-None]	8.9349ms	8.5414ms	117.0767 Ops/s	115.9305 Ops/s	$\color{#35bf28}+0.99\%$
test_iql_speed[True-backward]	17.3045ms	16.7706ms	59.6282 Ops/s	58.9340 Ops/s	$\color{#35bf28}+1.18\%$
test_iql_speed[reduce-overhead-None]	9.0647ms	8.5759ms	116.6065 Ops/s	114.8342 Ops/s	$\color{#35bf28}+1.54\%$
test_iql_speed[reduce-overhead-backward]	17.5872ms	17.1900ms	58.1734 Ops/s	56.5607 Ops/s	$\color{#35bf28}+2.85\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.3395ms	6.0577ms	165.0793 Ops/s	165.8725 Ops/s	$\color{#d91a1a}-0.48\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.5722ms	0.3534ms	2.8300 KOps/s	3.5624 KOps/s	$\textbf{\color{#d91a1a}-20.56\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6141ms	0.3092ms	3.2343 KOps/s	3.8485 KOps/s	$\textbf{\color{#d91a1a}-15.96\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	6.0226ms	5.7540ms	173.7922 Ops/s	175.9763 Ops/s	$\color{#d91a1a}-1.24\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.7087ms	0.3503ms	2.8549 KOps/s	3.1487 KOps/s	$\textbf{\color{#d91a1a}-9.33\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.8367ms	0.3369ms	2.9678 KOps/s	3.0850 KOps/s	$\color{#d91a1a}-3.80\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	1.5664ms	1.3673ms	731.3665 Ops/s	761.0467 Ops/s	$\color{#d91a1a}-3.90\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	1.7300ms	1.2861ms	777.5598 Ops/s	793.2859 Ops/s	$\color{#d91a1a}-1.98\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	9.3553ms	6.0104ms	166.3784 Ops/s	169.6720 Ops/s	$\color{#d91a1a}-1.94\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.8850ms	0.5198ms	1.9238 KOps/s	2.1507 KOps/s	$\textbf{\color{#d91a1a}-10.55\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8723ms	0.4097ms	2.4410 KOps/s	2.2086 KOps/s	$\textbf{\color{#35bf28}+10.52\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	5.8687ms	5.7219ms	174.7677 Ops/s	174.1772 Ops/s	$\color{#35bf28}+0.34\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	2.4655ms	0.3329ms	3.0035 KOps/s	2.6833 KOps/s	$\textbf{\color{#35bf28}+11.93\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.5896ms	0.3319ms	3.0130 KOps/s	3.1326 KOps/s	$\color{#d91a1a}-3.82\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.8870ms	5.6874ms	175.8284 Ops/s	174.4918 Ops/s	$\color{#35bf28}+0.77\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.1247ms	0.2781ms	3.5958 KOps/s	2.7274 KOps/s	$\textbf{\color{#35bf28}+31.84\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.4896ms	0.2556ms	3.9129 KOps/s	2.8585 KOps/s	$\textbf{\color{#35bf28}+36.89\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	6.0683ms	5.8645ms	170.5169 Ops/s	168.7704 Ops/s	$\color{#35bf28}+1.03\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.8086ms	0.4328ms	2.3108 KOps/s	1.8661 KOps/s	$\textbf{\color{#35bf28}+23.83\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8264ms	0.4126ms	2.4239 KOps/s	1.9348 KOps/s	$\textbf{\color{#35bf28}+25.28\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	6.6702ms	5.0679ms	197.3197 Ops/s	195.8593 Ops/s	$\color{#35bf28}+0.75\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	5.5689ms	2.2582ms	442.8229 Ops/s	408.6129 Ops/s	$\textbf{\color{#35bf28}+8.37\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	9.2326ms	1.2056ms	829.4693 Ops/s	946.6985 Ops/s	$\textbf{\color{#d91a1a}-12.38\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.5102s	15.1943ms	65.8143 Ops/s	191.0643 Ops/s	$\textbf{\color{#d91a1a}-65.55\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	8.9872ms	1.9339ms	517.0874 Ops/s	430.0544 Ops/s	$\textbf{\color{#35bf28}+20.24\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	2.2833ms	1.0436ms	958.2631 Ops/s	812.4425 Ops/s	$\textbf{\color{#35bf28}+17.95\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	7.1892ms	5.2720ms	189.6826 Ops/s	54.9794 Ops/s	$\textbf{\color{#35bf28}+245.01\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	8.9135ms	2.1971ms	455.1370 Ops/s	470.4391 Ops/s	$\color{#d91a1a}-3.25\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	7.6521ms	1.3670ms	731.5151 Ops/s	977.5762 Ops/s	$\textbf{\color{#d91a1a}-25.17\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	34.7987ms	32.7132ms	30.5687 Ops/s	29.6130 Ops/s	$\color{#35bf28}+3.23\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	18.9949ms	17.5262ms	57.0573 Ops/s	55.1067 Ops/s	$\color{#35bf28}+3.54\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True]	0.4566s	42.0204ms	23.7980 Ops/s	28.7641 Ops/s	$\textbf{\color{#d91a1a}-17.27\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	19.7311ms	17.8196ms	56.1178 Ops/s	54.2211 Ops/s	$\color{#35bf28}+3.50\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	37.7164ms	35.4181ms	28.2341 Ops/s	27.6234 Ops/s	$\color{#35bf28}+2.21\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	20.8534ms	19.3100ms	51.7866 Ops/s	50.7718 Ops/s	$\color{#35bf28}+2.00\%$