fix: chat_template loading and spatial_shapes unbound variable in Eagle VL#582
Open
cagataycali wants to merge 4 commits intoNVIDIA:mainfrom
Open
fix: chat_template loading and spatial_shapes unbound variable in Eagle VL#582cagataycali wants to merge 4 commits intoNVIDIA:mainfrom
cagataycali wants to merge 4 commits intoNVIDIA:mainfrom
Conversation
transformers >=4.51 requires chat_template to be set on the processor for apply_chat_template() to work, but AutoProcessor.from_pretrained() does not always load it from the bundled Eagle module's chat_template.json or tokenizer_config.json. This adds a fallback in build_processor() that reads the template from disk if the processor's chat_template is None after loading.
When select_layer != -1, the else branch in extract_feature() never captured spatial_shapes from vision_model_output before using it in pixel_shuffle_back(). Initialize spatial_shapes = None and extract it from the vision model output in both branches. Also ensures chat_template.json is loaded as the multimodal-aware template (not the simpler tokenizer_config.json one).
cagataycali
commented
Mar 18, 2026
cagataycali
commented
Mar 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two bug fixes for GR00T N1.5/N1.6 inference with
transformers>=4.51:Bug 1:
chat_templateisNone→ValueError: Cannot render chat templateFile:
gr00t/model/gr00t_n1d6/processing_gr00t_n1d6.pyRoot cause:
AutoProcessor.from_pretrained()loadstokenizer_config.jsonwhich has a simpler template without<image>token support. The proper multimodal template lives inchat_template.jsonbut is not loaded by the processor.Fix: After
AutoProcessor.from_pretrained(), check ifchat_templateisNone. If so, load it fromchat_template.json(or fall back totokenizer_config.json) in the Eagle model directory.Bug 2:
spatial_shapesunbound →UnboundLocalErrorFile:
gr00t/model/modules/nvidia/Eagle-Block2A-2B-v2/modeling_eagle3_vl.pyRoot cause: In
extract_feature(), whenselect_layer != -1(GR00T uses-4), theelsebranch only grabshidden_states[select_layer]but never capturesspatial_shapesfromvision_model_output. Line 366 then usesspatial_shapeswhich was never assigned.Fix: Initialize
spatial_shapes = Nonebefore the branch, and extract it fromvision_model_outputin bothifandelsebranches.Reproduction
Testing
Verified with 12 integration tests on an NVIDIA L40S GPU (Python 3.10,
transformers==4.51.3,gr00t==0.1.0). All pass after these fixes.Changes
processing_gr00t_n1d6.pychat_template.jsonfallback whenAutoProcessormisses itmodeling_eagle3_vl.pyspatial_shapes = None; extract in both branchesAdditional note
chat_template.jsonexists in the source tree atgr00t/model/modules/nvidia/Eagle-Block2A-2B-v2/chat_template.jsonbut is not included in the pip-installed wheel (gr00t==0.1.0). Users installing via pip need to manually copy this file. Consider adding it to the wheel packaging.