Errors in simulation & problems reproducing the results from the checkpoints

[DanielDauner]

Hi, Thanks a lot for proving your code and checkpoints! Congrats on the strong results.

Unfortunately, I am facing issues reproducing the results on Val14 with the provided checkpoints. The results are so low, that I expect a mistake at some point of me running your code. E.g. the results for cpks800m are:

	CLS	NAC	DAC	DDC	EIMP	TTC	EPAR	SLC	EIC
reactive	0.444323	0.838104	0.728086	0.978980	0.758497	0.707513	0.597952	0.999285	0.477639
non-reactive	0.561902	0.826029	0.788014	0.988819	0.834526	0.693202	0.655528	0.999403	0.544723

What I did was (1) downloaded the checkpoints from hugging face, (2) installed the code according to the README.md, (3) and executed:

MODEL_PATH="/path/to/str_open/cpks800m"
SPLIT_FILTER_YAML="/path/to/val14_split.yaml"
DATA_PATH=/path/to/trainval/

python run_simulation_closed.py \
    --data_path ${DATA_PATH} \
    --map_path ${NUPLAN_MAPS_ROOT} \
    --model_path ${MODEL_PATH} \
    --split_filter_yaml ${SPLIT_FILTER_YAML} \
    --initstable_time 8 \
    --conservative_factor 0.8 \
    --comfort_weight 10.0 \
    --batch_size 7 \
    --processes-repetition 5 \
    --test_type 'closed_loop_nonreactive_agents' \
    --nuplan_exp_root "/path/to/exp" \
    --pdm_lateral_offsets none;

Then, I got this error:

Traceback (most recent call last):
  File ".../StateTransformer/run_simulation_closed.py", line 652, in __call__
    result = (True, _worker_func(*task, self.model, self.args))
  File ".../StateTransformer/run_simulation_closed.py", line 744, in _worker_func
    reports_tmp = simulation_runner.run()
  File ".../StateTransformer/run_simulation_closed.py", line 323, in run
    states_in_batch, scores_in_batch = self.planners[0]._str_generator.predict_states_in_batch(
  File ".../StateTransformer/nuplan_simulation/str_trajectory_generator.py", line 147, in predict_states_in_batch
    trajectories, scores = self._compute_str_trajectory(
  File ".../StateTransformer/nuplan_simulation/str_trajectory_generator.py", line 182, in _compute_str_trajectory
    prediction_generation = self._model.generate(
  File ".../StateTransformer/run_simulation_closed.py", line 664, in _generate
    raise e
  File ".../StateTransformer/run_simulation_closed.py", line 662, in _generate
    return func(*args, **kwargs)
  File ".../miniconda3/envs/str/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File ".../StateTransformer/transformer4planning/models/backbone/str_base.py", line 498, in generate
    trajectory_label_dummy = torch.zeros((batch_size, pred_length, 4), device=device)
UnboundLocalError: local variable 'pred_length' referenced before assignment

I tried fixing it by adding pred_length = info_dict.get("pred_length", 0) at this line, but then the code and checkpoint produces bad results. Could you help me fix this error and reproduce the results? Thanks a lot for your help!!

Best,
Daniel

[larksq]

Hi Daniel,

Thank you for your kind words and for taking the time to detail the issue you're encountering. I’m sorry that you’re having trouble reproducing the results on Val14 with the provided checkpoints.

First and formost, it is not expected to have the error about undefined pred_length. For the checkpoint you downloaded, the use_proposal at the config should not be False or 0. This should lead to the code block of defining the pred_length at Line 432. The config should be printed when the model is initialized. Could you follow these steps and check if:

1. Used the checkpoint of 'cpks800m' for closed-loop reproduction.
2. Check the log at initialization to ensure the use_proposal is properly loaded. The value should be 512.

If you correctly loaded the checkpoint and init the model, this error should be gone. You can run the test14hard set (considering the size) for a faster sanity check, the score should not be lower than 80.

Best,
Qiao