use tok as fallback tokenizer#11

kootenpv wants to merge 1 commit intosoaxelbrooke:mastersoaxelbrooke/python-bpe:masterfrom

kootenpv:masterkootenpv/python-bpe:masterCopy head branch name to clipboard