How to convert FP16 or FP8 model files to NVFP4 model files

[Alan-D-Chen]

Hello, first of all, I would like to express my gratitude for your tremendous contributions to the field of LLM, which have driven significant advancements in the TensorRT-LLM framework. I am a computer science student who is using the TensorRT-LLM framework for the first time. I have some questions (regarding how to convert FP16 or FP8 model files to NVFP4 model files) that I would like to ask all of you.
First, I have read the following information:
For the basic environment, I have already created a container using the relevant command. (I learned that the conversion can be accomplished using https://github.com/NVIDIA/Model-Optimizer or the TensorRT-LLM framework at https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags?version=1.2.0rc2.post1.)

docker run -it --name tensor-llm-alanchen --ipc host --gpus all --ulimit memlock=-1 --ulimit stack=67108864 -p 8000:8000 -v /data2:/data2 -v /data3:/data3 -v /data4:/data4 nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc4

Could you please tell me the specific steps to convert FP16 or FP8 model files to NVFP4 model files? Alternatively, are there any easy-to-read tutorials that I can refer to?

[Alan-D-Chen]

I can locate the FP16 or FP8 model files of the DeepSeek series and Qwen3 series in my container, for example:

root@e86133b171db:/data3/Qwen3-32B-FP8# ls
README.md                         model-00004-of-00007.safetensors
config.json                       model-00005-of-00007.safetensors
configuration.json                model-00006-of-00007.safetensors
generation_config.json            model-00007-of-00007.safetensors
merges.txt                        model.safetensors.index.json
model-00001-of-00007.safetensors  tokenizer.json
model-00002-of-00007.safetensors  tokenizer_config.json
model-00003-of-00007.safetensors  vocab.json
root@e86133b171db:/data3/Qwen3-32B-FP8# pwd
/data3/Qwen3-32B-FP8
root@e86133b171db:/data3/Qwen3-32B-FP8# 

How should I convert these model files into NVFP4 model files?

[Alan-D-Chen]

I have read these files，but they are too much hard to understand them.
https://nvidia.github.io/Model-Optimizer/guides/1_quantization.html
https://nvidia.github.io/Model-Optimizer/guides/_choosing_quant_methods.html
https://nvidia.github.io/Model-Optimizer/guides/_pytorch_quantization.html
https://nvidia.github.io/Model-Optimizer/reference/generated/modelopt.torch.quantization.config.html#quantization-formats
https://nvidia.github.io/Model-Optimizer/guides/_customized_model_quantization.html
https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/onnx_ptq

@shengliangxu Would you like to do me a favor to garde me to convert FP16 or FP8 model files to NVFP4 model files ?
@ajrasane @kevalmorabia97

[Alan-D-Chen]

I get some help for AI agent , like:

import torch
from tensorrt_llm import QuantMode
from tensorrt_llm.builder import Builder
from tensorrt_llm.models import LlamaForCausalLM  # DeepSeek R1 基于 Llama 架构适配
from transformers import AutoModelForCausalLM, AutoTokenizer

# 1. 加载原始 FP16/FP8 DeepSeek R1 模型（从 Hugging Face 下载）
model_name = "deepseek-ai/DeepSeek-R1-7B-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
# 加载 FP16 模型（若为 FP8 需指定 torch_dtype=torch.float8_e4m3fn）
hf_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="cuda:0"
)

# 2. 配置 NVFP4 量化参数（关键）
quant_mode = QuantMode(
    quantize_weights=True,  # 量化权重
    weight_quantization_bits=4,  # 4位量化
    weight_quantization_scheme="nvfp4",  # 指定 NVFP4 格式
    quantize_activations=False,  # NVFP4 通常仅量化权重，激活仍用 FP16/FP8
    calibration_mode=False  # NVFP4 权重量化无需校准数据（重点！）
)

# 3. 构建 TensorRT-LLM 引擎（转换 + 编译）
builder = Builder()
# 将 Hugging Face 模型转换为 TensorRT-LLM 模型并应用 NVFP4 量化
trt_model = LlamaForCausalLM.from_hugging_face(
    hf_model,
    tokenizer,
    quant_mode=quant_mode
)

# 4. 编译生成 NVFP4 格式的 TensorRT 引擎文件（.plan）
engine = builder.build(trt_model)
# 保存 NVFP4 模型文件

engine.save("deepseek_r1_7b_nvfp4.engine")

BUT, these lines does not work at all ~~~

[elizabetht]

@Alan-D-Chen Recently, I was able to use Model-Optimizer in PTQ mode and get FP16 to NVFP4 conversion working. Here is my notebook that worked: https://github.com/elizabetht/language-modeling-from-scratch/blob/main/quantization/model-optimizer/NVFP4_PTQ_Min-Max_Calibration.ipynb

[elizabetht]

For FP16 to FP8, here is the notebook link: https://github.com/elizabetht/language-modeling-from-scratch/blob/main/quantization/model-optimizer/FP8_PTQ_Min-Max_Calibration.ipynb. Let me know if you have any questions @Alan-D-Chen

[Alan-D-Chen]

@elizabetht THANKS you so much ! I have got Qwen3-8B-BF16 to Qwen3-8B-NVFP4 under TensorRT LLM ([nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc4](http://nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc4))

It does work.
AND Thank you again.

[jenchen13]

Hello please try the hf_ptq.py example here https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq#quantize-and-export

If your issue resolved can I close this?