Script to push the trained models and tokenizer to HF#45
Merged
chandanms merged 4 commits intosimple-stories:devfrom Sep 15, 2025
Merged
Script to push the trained models and tokenizer to HF#45chandanms merged 4 commits intosimple-stories:devfrom
chandanms merged 4 commits intosimple-stories:devfrom
Conversation
chandanms
changed the title
danbraunai
approved these changes
Sep 1, 2025
Collaborator
danbraunai
left a comment
danbraunai
left a comment
There was a problem hiding this comment.
The changes made look good. Though I thought you'd want to update the push_to_hf.py script so that it uploads all the nice information that you previously manually added. Notably, the README that you see at e.g. https://huggingface.co/SimpleStories/SimpleStories-1.25M.
You can do that in a new PR, though you probably want to just do it here.
scripts/push_to_hf.py
Outdated
| """Try to resolve a tokenizer file path from the final_config.yaml next to the checkpoint. | ||
| def find_saved_tokenizer(output_dir: Path) -> Path | None: | ||
| """Find the saved tokenizer in the training output directory.""" | ||
| # Look for tokenizer.json saved during training |
scripts/push_to_hf.py
Outdated
| return None | ||
|
|
||
| # As a last resort, if the file name matches a known tokenizer in the repo, use it | ||
| # If not found, return default tokenizer |
…e HF models upload
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Script to push the models and tokenizer to HF. Stores the tokenizer during training and uses that to convert to HF and push to HF.
Related Issue
Closes #42
Motivation and Context
Makes it easier to upload the model and tokenizer to HF. The accompanying tokenizer is saved during training rather than pointing the path to it later to upload to HF.
How Has This Been Tested?
Added a test case to test for conversion of tokenizer to HF compatible format
Does this PR introduce a breaking change?
No