Megatron-LM specific release v2.8 rocm cherrypicks

[sudhu2k]

Description

These cherry picks fixes some issues that's seen in Megatron-LM unit test runs.

Type of change

• Documentation change (change only to the documentation, either a fix or a new content)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Infra/Build change
• Code refactoring

Checklist:

• I have read and followed the contributing guidelines
• The functionality is complete
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[ipanfilo]

It requires test hotfix doesn't it?

[sudhu2k]

This one didn't require the hotfix.

TE2.10 required hotfix:

TransformerEngine/tests/pytorch/distributed/test_cast_master_weights_to_fp8.py

Lines 722 to 724 in 3344b85

	# ROCm: Use executable as-is; do not resolve() or a venv symlink may point to system
	# Python which does not have torch/site-packages.
	python_exe = pathlib.Path(sys.executable)

TE2.8 implements the test differently:

TransformerEngine/tests/pytorch/distributed/test_cast_master_weights_to_fp8.py

Lines 27 to 40 in a25a454

	def _run_test(quantization):
	test_path = TEST_ROOT / "run_cast_master_weights_to_fp8.py"
	test_cmd = LAUNCH_CMD + [str(test_path)] + ["--quantization", quantization]
	result = subprocess.run(test_cmd, env=os.environ, check=False)
	assert result.returncode == 0
	@pytest.mark.parametrize("quantization", ["fp8", "fp8_cs", "fp8_block"])
	def test_cast_master_weights_to_fp8(quantization):
	if quantization in ("fp8", "fp8_cs") and not fp8_available:
	pytest.skip(reason_for_no_fp8)
	if quantization == "fp8_block" and not fp8_block_scaling_available:
	pytest.skip(reason_for_no_fp8_block_scaling)
	_run_test(quantization)

[ipanfilo]

Run CI level 3

[sudhu2k]

Run CI level 3

MGPU/SGPU tests passed on level 3
failed on example test on MI325 with a flaky huggingface error:

File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 3010, in dataset_info
   hf_raise_for_status(r)
 File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 880, in hf_raise_for_status
   raise _format(HfHubHTTPError, message, response) from e
huggingface_hub.errors.HfHubHTTPError: (Request ID: Root=1-69bcf50c-102ad0297e9cea061658671b;35076563-ba85-4f92-a0a7-3f465e541e0a)

429 Too Many Requests: you have reached your 'api' rate limit.

https://github.com/ROCm/TransformerEngine/actions/runs/23328907485/job/67856033151