[Bug]: Qwen3-VL fails during multimodal encoder profiling (expected 3 dims, got 2) on Blackwell + NVFP4 (FlashInfer) even after CUDA header fix

[Firworksyt]

Your current environment

The output of python collect_env.py

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.9.0+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.12 (main, Oct 10 2025, 08:52:57) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-5.15.0-161-generic-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   :
GPU models and configuration : GPU 0: NVIDIA RTX PRO 6000 Blackwell Server Edition
Nvidia driver version        : 580.95.05
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           52 bits physical, 57 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  30
On-line CPU(s) list:                     0-29
Vendor ID:                               AuthenticAMD
Model name:                              AMD EPYC 9555 64-Core Processor
CPU family:                              26
Model:                                   2
Thread(s) per core:                      1
Core(s) per socket:                      1
Socket(s):                               30
Stepping:                                1
BogoMIPS:                                6399.99
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 clzero xsaveerptr wbnoinvd arat npt lbrv nrip_save tsc_scale vmcb_clean flushbyasid pausefilter pfthreshold v_vmsave_vmload vgif avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid movdiri movdir64b fsrm avx512_vp2intersect flush_l1d arch_capabilities
Virtualization:                          AMD-V
Hypervisor vendor:                       KVM
Virtualization type:                     full
L1d cache:                               1.9 MiB (30 instances)
L1i cache:                               1.9 MiB (30 instances)
L2 cache:                                15 MiB (30 instances)
L3 cache:                                480 MiB (30 instances)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-29
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP disabled; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.5.3
[pip3] numpy==2.2.0
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.16.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.3.1
[pip3] nvidia-ml-py==13.580.82
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.3.20
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] torch==2.9.0+cu129
[pip3] torchaudio==2.9.0+cu129
[pip3] torchvision==0.24.0+cu129
[pip3] transformers==4.57.3
[pip3] triton==3.5.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.11.2.dev347+gc7ba1f6bc (git sha: c7ba1f6bc)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-29    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=void
NVIDIA_REQUIRE_CUDA=cuda>=12.9 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.9.1
LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64
NVIDIA_CTK_LIBCUDA_DIR=/usr/lib/x86_64-linux-gnu
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1

🐛 Describe the bug

When serving an NVFP4-quantized Qwen3-VL model on a Blackwell GPU using the vllm-openai:nightly image, engine initialization fails during the multimodal encoder profiling step with:

> ValueError: not enough values to unpack (expected 3, got 2)

This happens after weights load successfully. The failure occurs inside the Qwen2.5-VL attention block where it expects a [seq_len, batch_size, hidden_dim] tensor but receives a 2D tensor instead.

This is my command to reproduce the issue:
docker run --runtime nvidia --gpus all --ipc=host -p 8000:8000
-e TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
-v /usr/local/cuda-12.8/targets/x86_64-linux/include:/usr/local/cuda/include:ro
vllm/vllm-openai:nightly
Firworks/Qwen3-VL-32B-Thinking-nvfp4
--dtype auto
--max-model-len 32768
--gpu-memory-utilization 0.9
--limit-mm-per-prompt.video 0

Model downloads and loads successfully.

Failure happens during engine startup, before any client request.

--limit-mm-per-prompt.video 0 is set; I was getting an additional error about the video profiler but I set this to try to get past that. There may be an issue there but if I could at least test text and video on this model that would still be progress.

The actual failure trace is:

File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 4277, in profile_run
    dummy_encoder_outputs = self.model.embed_multimodal(
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py", line 1503, in embed_multimodal
    image_embeddings = self._process_image_input(multimodal_input)
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py", line 1382, in _process_image_input
    image_embeds = self.visual(pixel_values, grid_thw=grid_thw)
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py", line 571, in forward
    hidden_states = blk(
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2_5_vl.py", line 380, in forward
    seq_len, batch_size, _ = x.shape
ValueError: not enough values to unpack (expected 3, got 2)

From the error, x.shape is only 2D at this point (likely [N, D]), but the Qwen2.5-VL attention block assumes a 3D [seq_len, batch_size, hidden_dim] layout.

I'd be happy to try any workarounds or patches if anyone has any suggestions of something to try.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

To rule out CUDA graph issues, can you try --enforce-eager? cc @ywang96 @Isotr0py

[ywang96]

I'm not sure if this is an issue specific to NVFP4 but FWIW I was able to launch the model server on H100 with the following command on main branch

vllm serve Qwen/Qwen3-VL-32B-Thinking -tp 4

[Firworksyt]

I'm not sure if this is an issue specific to NVFP4 but FWIW I was able to launch the model server on H100 with the following command on main branch

vllm serve Qwen/Qwen3-VL-32B-Thinking -tp 4

I would not at all be surprised if it were related to NVFP4. Everything still feels pretty fragile with quantizing and running NVFP4.

To rule out CUDA graph issues, can you try --enforce-eager? cc @ywang96 @Isotr0py

I tried rerunning with --enforce-eager on the latest nightly image and I’m still seeing the same error.

docker run --runtime nvidia --gpus all --ipc=host -p 8000:8000 -e TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 -v /usr/local/cuda-12.8/targets/x86_64-linux/include:/usr/local/cuda/include:ro vllm/vllm-openai:nightly Firworks/Qwen3-VL-32B-Thinking-nvfp4 --dtype auto --max-model-len 32768 --gpu-memory-utilization 0.9 --limit-mm-per-prompt.video 0 --enforce-eager

I do see this making me think the flag took effect:
(APIServer pid=1) INFO 11-29 11:43:31 [vllm.py:695] Cudagraph is disabled under eager mode

Traceback:

File "vllm/v1/worker/gpu_model_runner.py", line 4280, in profile_run
  dummy_encoder_outputs = self.model.embed_multimodal(...)
File "vllm/model_executor/models/qwen3_vl.py", line 1512, in embed_multimodal
  image_embeddings = self._process_image_input(multimodal_input)
File "vllm/model_executor/models/qwen3_vl.py", line 1391, in _process_image_input
  image_embeds = self.visual(pixel_values, grid_thw=grid_thw)
File "vllm/model_executor/models/qwen3_vl.py", line 560, in forward
  hidden_states = blk(hidden_states, grid_thw)
File "vllm/model_executor/models/qwen3_vl.py", line 237, in forward
  x = x + self.attn(...)
File "vllm/model_executor/models/qwen2_5_vl.py", line 368, in forward
  seq_len, batch_size, _ = x.shape
ValueError: not enough values to unpack (expected 3, got 2)

Any other ideas for things to try or patch?

[Isotr0py]

@Firworksyt Can you check if #29742 fixes this issue? Thanks!

[Firworksyt]

Thank you for the quick fix! I applied the changes from #29742 to the compressed_tensors_w4a4_nvfp4.py file inside the vllm/vllm-openai:nightly image by bind-mounting the updated file over the existing one:

docker run --runtime nvidia --gpus all --ipc=host -p 8000:8000 \
  -e TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 \
  -v /usr/local/cuda-12.8/targets/x86_64-linux/include:/usr/local/cuda/include:ro \
  -v ~/vllm_patch/compressed_tensors_w4a4_nvfp4.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4.py:ro \
  vllm/vllm-openai:nightly \
  Firworks/Qwen3-VL-32B-Thinking-nvfp4 \
  --dtype auto \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.9 \
  --limit-mm-per-prompt.video 0 \
  --enforce-eager

With this patch in place, the original ValueError: not enough values to unpack (expected 3, got 2) is gone, but engine initialization still fails with a new error from the FP4 GEMM path:

RuntimeError: [FP4 gemm Runner] FP4 Gemm cutlass kernel will fail for params. Error: Invalid status

  File "vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4.py", line 202, in apply_weights
    out = flashinfer_scaled_fp4_mm(*mm_args, backend=backend_name)
  File "vllm/utils/flashinfer.py", line 462, in flashinfer_scaled_fp4_mm
    return flashinfer_mm_fp4(
  ...
  File "flashinfer/gemm/gemm_base.py", line 486, in cutlass_fp4_gemm
    fp4_runner(inputs=inputs, tactic=tactic)
  File "flashinfer/autotuner.py", line 217, in __call__
    return self.forward(inputs, **kwargs)
  File "flashinfer/gemm/gemm_base.py", line 428, in forward
    module.fp4_gemm(
  File "python/tvm_ffi/cython/function.pxi", line 904, in tvm_ffi.core.Function.__call__
  RuntimeError: [FP4 gemm Runner] FP4 Gemm cutlass kernel will fail for params. Error: Invalid status

From the stack trace this is happening in an MLP linear:

vllm/model_executor/models/qwen3_vl.py", line 245, in forward
  x = x + self.mlp(self.norm2(x))
vllm/model_executor/layers/linear.py", line 1405, in forward
  output_parallel = self.quant_method.apply(self, input_parallel, bias_)
vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py", line 916, in apply
  return scheme.apply_weights(layer, x, bias=bias)

Let me know if there’s any environment variable or config I can try to force a different FP4 GEMM backend / fallback, or if you’d like me to grab any additional debug logs. I did try adding -e VLLM_FLASHINFER_ENABLED=0 just as a test to see where it would go but it also bombed out:

RuntimeError: [FP4 gemm Runner] FP4 Gemm cutlass kernel will fail for params. Error: Invalid status

So anyhow I think the bugfix did work because I got further along but it seems like there are still more problems to figure out before this can be run. Is there any more information you'd like from me or additional tests to try?

[Isotr0py]

File "flashinfer/gemm/gemm_base.py", line 486, in cutlass_fp4_gemm
fp4_runner(inputs=inputs, tactic=tactic)
File "flashinfer/autotuner.py", line 217, in call
return self.forward(inputs, **kwargs)
File "flashinfer/gemm/gemm_base.py", line 428, in forward
module.fp4_gemm(
File "python/tvm_ffi/cython/function.pxi", line 904, in tvm_ffi.core.Function.call
RuntimeError: [FP4 gemm Runner] FP4 Gemm cutlass kernel will fail for params. Error: Invalid status

Seems that this is an issue about flashinfer kernel's tvm_ffi during JIT compilation, can you try to switch to pre-compiled pure cutlass kernel through VLLM_NVFP4_GEMM_BACKEND="cutlass"?

[Firworksyt]

@Isotr0py I tried with that backend:

docker run --runtime nvidia --gpus all --ipc=host -p 8000:8000 \
  -e TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 \
  -e VLLM_NVFP4_GEMM_BACKEND="cutlass" \
  -v /usr/local/cuda-12.8/targets/x86_64-linux/include:/usr/local/cuda/include:ro \
  -v ~/vllm_patch/compressed_tensors_w4a4_nvfp4.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4.py:ro \
  vllm/vllm-openai:nightly \
  Firworks/Qwen3-VL-32B-Thinking-nvfp4 \
  --dtype auto \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.9 \
  --limit-mm-per-prompt.video 0 \
  --enforce-eager

And then got the following error:

File ".../compressed_tensors_w4a4_nvfp4.py", line 214, in apply_weights
  out = cutlass_scaled_fp4_mm(*mm_args)
File "/usr/local/lib/python3.12/dist-packages/vllm/_custom_ops.py", line 772, in cutlass_scaled_fp4_mm
  torch.ops._C.cutlass_scaled_fp4_mm(...)
...
RuntimeError: Error Internal

I don't know if it will help at all but I instrumented where it failed a bit and reran it:

        else:
            assert self.backend == "cutlass"
            try:
                out = cutlass_scaled_fp4_mm(*mm_args)
            except Exception as e:
                # debug info
                print("[NVFP4 debug] backend:", self.backend)
                print("[NVFP4 debug] output_dtype:", output_dtype)

                print("[NVFP4 debug] x_fp4:",
                      "shape", tuple(x_fp4.shape),
                      "dtype", x_fp4.dtype,
                      "device", x_fp4.device)
                print("[NVFP4 debug] weight_packed:",
                      "shape", tuple(layer.weight_packed.shape),
                      "dtype", layer.weight_packed.dtype,
                      "device", layer.weight_packed.device)
                print("[NVFP4 debug] x_blockscale:",
                      "shape", tuple(x_blockscale.shape),
                      "dtype", x_blockscale.dtype,
                      "device", x_blockscale.device)
                print("[NVFP4 debug] weight_scale:",
                      "shape", tuple(layer.weight_scale.shape),
                      "dtype", layer.weight_scale.dtype,
                      "device", layer.weight_scale.device)
                print("[NVFP4 debug] alpha:",
                      layer.alpha, "dtype", layer.alpha.dtype)

                # GPU / arch info if available
                if x_fp4.is_cuda:
                    import torch
                    dev = x_fp4.device
                    idx = 0 if dev.index is None else dev.index
                    print("[NVFP4 debug] cuda_device_index:", idx)
                    print("[NVFP4 debug] cuda_device_name:",
                          torch.cuda.get_device_name(idx))
                    print("[NVFP4 debug] cuda_device_capability:",
                          torch.cuda.get_device_capability(idx))

                print("[NVFP4 debug] original exception type:", type(e).__name__)
                print("[NVFP4 debug] original exception:", e)
                # Re-raise so the behavior is unchanged
                raise

        if bias is not None:
            out = out + bias
        return out.view(*output_shape)

[NVFP4 debug] backend: cutlass
[NVFP4 debug] output_dtype: torch.bfloat16
[NVFP4 debug] x_fp4:
  shape (65536, 576)
  dtype torch.uint8
  device cuda:0
[NVFP4 debug] weight_packed:
  shape (3456, 576)
  dtype torch.uint8
  device cuda:0
[NVFP4 debug] x_blockscale:
  shape (65536, 72)
  dtype torch.float8_e4m3fn
  device cuda:0
[NVFP4 debug] weight_scale:
  shape (3456, 72)
  dtype torch.float8_e4m3fn
  device cuda:0
[NVFP4 debug] alpha:
  tensor(9.2145e-23, device='cuda:0', dtype=torch.float32)
[NVFP4 debug] cuda_device_index: 0
[NVFP4 debug] cuda_device_name: NVIDIA RTX PRO 6000 Blackwell Server Edition
[NVFP4 debug] cuda_device_capability: (12, 0)
[NVFP4 debug] original exception type: RuntimeError
[NVFP4 debug] original exception: Error Internal

[Isotr0py]

Hmmm, I also tried to initialize language backbone without ViT components on RTX5090, and it also failed to be initialized due to kernel error:

vllm serve /root/autodl-tmp/Qwen3-VL-32B-Thinking-nvfp4 --enforce-eager  --limit_mm_per_prompt.video 0 --limit_mm_per_pr
ompt.image 0

(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843] EngineCore failed to start.
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843] Traceback (most recent call last):
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/v1/engine/core.py", line 834, in run_engine_core
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/v1/engine/core.py", line 610, in __init__
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     super().__init__(
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/v1/engine/core.py", line 235, in _initialize_kv_caches
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/v1/executor/abstract.py", line 126, in determine_available_memory
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/v1/executor/uniproc_executor.py", line 75, in collective_rpc
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/v1/serial_utils.py", line 479, in run_method
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return func(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/miniconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return func(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/v1/worker/gpu_worker.py", line 324, in determine_available_memory
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     self.model_runner.profile_run()
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/v1/worker/gpu_model_runner.py", line 4359, in profile_run
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]                                         ^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/miniconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return func(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/v1/worker/gpu_model_runner.py", line 4073, in _dummy_run
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     outputs = self.model(
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]               ^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/model_executor/models/qwen3_vl.py", line 1651, in forward
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     hidden_states = self.language_model.model(
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/compilation/decorators.py", line 360, in __call__
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return self.forward(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/model_executor/models/qwen3_vl.py", line 1119, in forward
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     hidden_states, residual = layer(
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]                               ^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/model_executor/models/qwen3.py", line 221, in forward
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     hidden_states = self.self_attn(
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]                     ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/model_executor/models/qwen3.py", line 143, in forward
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     qkv, _ = self.qkv_proj(hidden_states)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/model_executor/layers/linear.py", line 565, in forward
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     output_parallel = self.quant_method.apply(self, input_, bias)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py", line 916, in apply
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return scheme.apply_weights(layer, x, bias=bias)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4.py", line 218, in apply_weights
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     out = cutlass_scaled_fp4_mm(*mm_args)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/autodl-tmp/vllm/vllm/_custom_ops.py", line 772, in cutlass_scaled_fp4_mm
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     torch.ops._C.cutlass_scaled_fp4_mm(out, a, b, block_scale_a, block_scale_b, alpha)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]   File "/root/miniconda3/lib/python3.12/site-packages/torch/_ops.py", line 1255, in __call__
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]     return self._op(*args, **kwargs)
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=10198) ERROR 12-01 14:22:49 [core.py:843] RuntimeError: Error Internal

@mgoin Perhaps there is some issues about the FP4 kenrel for sm100 device?

[Firworksyt]

If you tested it on an RTX5090 I think that's sm_120 same as the RTX Pro 6000 Blackwell I'm running right? sm_100 would be a B200, which, if it's helpful I could grab an instance and try this same thing on.

[ZJY0516]

I remember that cutlass_scaled_fp4_mm does not support sm120

[mgoin]

Can you please try #29711? There already is a cutlass_scaled_fp4_mm_sm120a but it seems it might not have been gated correctly if building multi-arch binaries