all 2 comments

sorted by: hot top controversial new old
[–] 1 point 6 months ago (1 child)

The chat does not use an actual tokenizer. It just counts characters then divides by 3 or 4, or something like that. The AI plugin, however, has an actual tokenizer for whatever AI chat model perchance is using. TLDR: it's just a rough estimate on token count. If it's consistently off, try altering the code and see if you can land at a better number. Otherwise, it's just a rough estimate.

  • source
  • hideshow 1 child comment