unnorm_key Assertion Error

[Boltzmachine]

I run the training script with SFT_MODEL_PATH="Haozhan72/Openvla-oft-SFT-libero10-trajall"

  File "/home/ubuntu/project/SimpleVLA-RL/verl/single_controller/ray/base.py", line 397, in func
    return getattr(self.worker_dict[key], name)(*args, **kwargs)
  File "/home/ubuntu/project/SimpleVLA-RL/verl/single_controller/base/decorator.py", line 404, in inner
    return func(*args, **kwargs)
  File "/home/ubuntu/project/SimpleVLA-RL/verl/workers/fsdp_workers.py", line 384, in init_model
    self.rollout, self.sharding_manager = self._build_rollout()
  File "/home/ubuntu/project/SimpleVLA-RL/verl/workers/fsdp_workers.py", line 318, in _build_rollout
    rollout = RobHFRollout(module=self.actor_module_fsdp, config=self.config.rollout)
  File "/home/ubuntu/project/SimpleVLA-RL/verl/workers/rollout/rob_rollout.py", line 235, in __init__
    self.vla_preprocess()
  File "/home/ubuntu/project/SimpleVLA-RL/verl/workers/rollout/rob_rollout.py", line 259, in vla_preprocess
    assert self.config.unnorm_key in self.module.norm_stats, f"Action un-norm key {unnorm_key} not found in VLA `norm_stats`!"
NameError: name 'unnorm_key' is not defined

[BroJunn]

I guess you don't set DATASET_NAME to "libero_10" but one of the others.

It seems that you can only use libero_10 if you want to use this ckpt SFT_MODEL_PATH="Haozhan72/Openvla-oft-SFT-libero10-trajall".

[zhan72]

Thank you for your attention!

Could you please provide more details, such as the specific configuration in run_openvla_oft_rl.sh?

As BroJunn mentioned, if you use a model that has been SFT on LIBERO-10 (such as "Haozhan72/Openvla-oft-SFT-libero10-trajall"), you need to specify the unnorm_key as libero_10.

We will provide more SFT models on LIBERO-Goal, Spatial, Object!

[Boltzmachine]

HI thanks for reply. I think I do set the DATASET_NAME to libro_10. Here is my script

set -x

export NCCL_DEBUG=WARN
export WANDB_API_KEY='459e3fc4a872232893a94bd2b74de2f2b4267df0'
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export TOKENIZERS_PARALLELISM=true
# export CUDA_LAUNCH_BLOCKING=1
export TORCH_USE_CUDA_DSA=1

PROJECT_NAME="SimpleVLA-RL"
EXPERIMENT_NAME='debug'
# For openvla-oft Libero-Long traj1 SFT or traj all SFT models can be find in https://huggingface.co/collections/Haozhan72/simplevla-rl-6833311430cd9df52aeb1f86
SFT_MODEL_PATH="Haozhan72/Openvla-oft-SFT-libero10-trajall"
CKPT_PATH="outputs/${PROJECT_NAME}/${EXPERIMENT_NAME}"
# DATASET_NAME can be libero_10 (libero_Long), libero_90, libero_spatial, libero_object, libero_goal
DATASET_NAME="libero_10"
VLA_NAME="openvla-oft"
NUM_GPUS=2
# If you want to use 2*8 GPU to RL. Set NUM_NODES=2
NUM_NODES=1
ALIGN_PATH="./align.json"

HYDRA_FULL_ERROR=1 python -m verl.trainer.main_ppo \
    data.task_suite_name=$DATASET_NAME \
    data.num_trials_per_task=50 \
    data.n_samples=8 \
    data.filter_accuracy=True \
    data.accuracy_lower_bound=0.1 \
    data.accuracy_upper_bound=0.9 \
    data.oversample_factor=1 \
    data.train_batch_size=16 \
    data.val_batch_size=128 \
    data.max_prompt_length=256 \
    data.max_response_length=128 \
    actor_rollout_ref.model.path=$SFT_MODEL_PATH \
    actor_rollout_ref.model.vla=$VLA_NAME \
    actor_rollout_ref.model.action_token_len=7 \
    actor_rollout_ref.model.action_chunks_len=8 \
    actor_rollout_ref.actor.optim.lr=5e-6 \
    actor_rollout_ref.actor.optim.warmup_style=constant \
    actor_rollout_ref.actor.ppo_mini_batch_size=2 \
    actor_rollout_ref.actor.ppo_micro_batch_size=$NUM_GPUS \
    actor_rollout_ref.actor.use_dynamic_bsz=False \
    actor_rollout_ref.actor.fsdp_config.param_offload=False \
    actor_rollout_ref.actor.fsdp_config.grad_offload=True \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \
    actor_rollout_ref.actor.grad_clip=1 \
    actor_rollout_ref.actor.clip_ratio_high=0.28 \
    actor_rollout_ref.actor.clip_ratio_low=0.2 \
    actor_rollout_ref.actor.num_images_in_input=1 \
    actor_rollout_ref.actor.traj_mini_batch_size=16 \
    actor_rollout_ref.model.enable_gradient_checkpointing=False \
    actor_rollout_ref.model.use_remove_padding=False \
    actor_rollout_ref.actor.entropy_coeff=0. \
    actor_rollout_ref.rollout.num_images_in_input=1 \
    actor_rollout_ref.rollout.val_micro_batch_size=8 \
    actor_rollout_ref.rollout.temperature=1.6 \
    actor_rollout_ref.rollout.experiment_name=$EXPERIMENT_NAME \
    actor_rollout_ref.rollout.micro_batch_size=1 \
    actor_rollout_ref.rollout.unnorm_key=$DATASET_NAME \
    actor_rollout_ref.rollout.model_family=openvla \
    actor_rollout_ref.rollout.task_suite_name=$DATASET_NAME \
    actor_rollout_ref.rollout.num_steps_wait=10 \
    actor_rollout_ref.rollout.pretrained_checkpoint=$SFT_MODEL_PATH \
    actor_rollout_ref.rollout.center_crop=True \
    actor_rollout_ref.rollout.max_prompt_length=512 \
    actor_rollout_ref.rollout.log_prob_micro_batch_size=32 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
    actor_rollout_ref.rollout.name=hf \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.9 \
    actor_rollout_ref.ref.log_prob_micro_batch_size=32 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    algorithm.kl_ctrl.kl_coef=0.00 \
    trainer.logger=['console','wandb'] \
    trainer.project_name=$PROJECT_NAME \
    trainer.experiment_name=$EXPERIMENT_NAME \
    trainer.default_local_dir=$CKPT_PATH/$PROJECT_NAME/$EXPERIMENT_NAME \
    trainer.n_gpus_per_node=$NUM_GPUS \
    trainer.nnodes=$NUM_NODES \
    trainer.save_freq=25 \
    trainer.test_freq=4 \
    trainer.total_epochs=100 \
    trainer.val_only=False \
    algorithm.adv_estimator=grpo \
    algorithm.adv_params.verifier_gamma=1.0 \
    algorithm.adv_params.reward_model_gamma=1.0 \
    trainer.runtime_env=$ALIGN_PATH \
    trainer.wandb_mode=online \
    trainer.val_before_train=True

[ranzhran]

I’m facing the same issue. Has anyone found a solution?

[zimgong]

#16 Hi folks, I patched this error by allowing loading model stats from the HF hub in this PR. Hope it fixes the issue.

[zhan72]

Fix for NameError: name 'unnorm_key' is not defined

Sorry for the late response. Regarding the NameError: name 'unnorm_key' is not defined error, the root cause is that when SFT_MODEL_PATH="Haozhan72/Openvla-oft-SFT-libero10-trajall", the model is downloaded from Hugging Face, but the path dataset_statistics_path = os.path.join(local_path, "dataset_statistics.json") becomes incorrect.

Solutions

There are two solutions to fix this issue:

Solution 1: Download the model locally first

1. Download "Haozhan72/Openvla-oft-SFT-libero10-trajall" from Hugging Face to your local machine
2. Set SFT_MODEL_PATH to point to the local path

Solution 2: Keep the model path and download dataset statistics separately

• Keep SFT_MODEL_PATH = "Haozhan72/Openvla-oft-SFT-libero10-trajall"
• Follow the approach in support load stats from hf hub #16
• Download dataset_statistics from Hugging Face first, then load it

Acknowledgments

Special thanks to zimgong for the pull request that helped resolve this issue.