🚀 Feature
I would like to be able to compute the number of tokens in a piece of text prior to embedding it.
Motivation
This tool would make it possible to know if the text will be truncated prior to being sent for embedding. If it was too long, one could summarize it first, chunk it, etc.
Basic example
count = token_count(text)
print(count)
# 57
Details
🚀 Feature
I would like to be able to compute the number of tokens in a piece of text prior to embedding it.
Motivation
This tool would make it possible to know if the text will be truncated prior to being sent for embedding. If it was too long, one could summarize it first, chunk it, etc.
Basic example
Details