Closed
Conversation
…and what pruning does
…omments for train and prune tokenizer
…in_config.yaml; deleted extra config.
Add pruning functionality to tokenizer creation to remove zero frequency tokens
* Add rms_norm_eps argument to config with default 1e-6 * Fix type issues * Add convert_to_hf.py script and tests * Re-enable tests in CI * Use torch>=2.6 in pyproject.toml * Improve naming * Fix type errors * Added conversion scripts and corresponding tests * Fixed pyright issues * Marked a test as slow since it downloads all models from HF * Revert "Marked a test as slow since it downloads all models from HF" This reverts commit 2c9aedb. Wrong commit with pytest! * Marked a test as slow since it downloads all models from HF * corrected the docstring of a test case. Made it more verbose to mention the backward compatibility --------- Co-authored-by: chandanms <mschandan96@gmail.com>
* Reset train dataloader when depleted * Fix pyright errors * Cast instead of isinstance * Update pinned torch version * Factor out gpt2 and make general train.py * Prefix wandb run name with model_id * Create gpt2 hf converters * Create push_to_hf * Upload tokenizer to hf too * Refactor gpt conversions
…ens; Made the data to tokenizer training iterable.
…_test_cases Tokenizer test cases and reformatting of tokenizer training file
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Updated readme. Fix for previous error in adding readme to wrong folder
Related Issue
Motivation and Context
How Has This Been Tested?
Does this PR introduce a breaking change?