feat(vllm): accept token lists and HuggingFace tokenizer results #589
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Support passing token IDs as list of ints or list of lists of ints - Support passing HuggingFace tokenizer output dicts (input_ids + attention_mask) - Filter padding tokens using attention_mask when provided - Fix deprecated vllm.transformers_utils.tokenizer import - Set pad_token to eos_token if not set - Add tests for token input compatibility 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Butanium
commented
Jan 7, 2026
Collaborator
Author
There was a problem hiding this comment.
fixes deprected imports
Butanium
commented
Jan 7, 2026
Collaborator
Author
Butanium
left a comment
Butanium
left a comment
There was a problem hiding this comment.
tests were claude generated, all code is mine
Member
Collaborator
Author
|
@JadenFiotto-Kaufman do you mean this is a bug introduced by this PR? |
Member
No I never got it working. It's on the list. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
input_ids+attention_mask)attention_maskwhen providedvllm.transformers_utils.tokenizerimportpad_tokentoeos_tokenif not set (mirroring LanguageModel behavior)Test plan
test_single_token_list- single list of token IDstest_batched_token_lists- multiple lists of token IDstest_hf_tokenizer_dict_single- HF tokenizer output for single prompttest_hf_tokenizer_dict_batched- HF tokenizer output for batched promptstest_hf_tokenizer_with_padding_mask- padding tokens filtered correctlytest_token_list_in_invoker- token list within invokertest_mixed_string_and_token_invokers- mixing string and token inputsNote:
test_invoker_group_batchingfails on upstream/main (pre-existing bug, unrelated to this PR)🤖 Generated with Claude Code