We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
There was an error while loading. Please reload this page.
1 parent 287ef1a commit 73d08eeCopy full SHA for 73d08ee
src/tiktoken.cpp
@@ -51,7 +51,7 @@ static Result<std::pair<std::string, uint64_t>> _parse(
51
// Tiktoken format
52
// https://github.com/openai/tiktoken/blob/main/tiktoken/load.py#L140 <base64
53
// encoded token str> <rank>
54
- auto pos = line.find(" ");
+ auto pos = line.find(' ');
55
TK_CHECK_OR_RETURN_ERROR(
56
pos != std::string::npos,
57
ParseFailure,
0 commit comments