Exploring and benchmarking multiple Byte Pair Encoding (BPE) implementations to build the fastest possible BPE tokenizer.
References : https://jytan.net/blog/2025/bpe/
-
Notifications
You must be signed in to change notification settings - Fork 0
ShlokVFX/Fast-BPE
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
Exploring all the various Bpe implementation to run fastest BPE tokenizer
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published