you are viewing a single comment's thread
view the rest of the comments
[–] 6 points 8 months ago

Here's an OpenAI page that allows you to enter text and see how it gets tokenized:

https://platform.openai.com/tokenizer

  • source
  • parent