you are viewing a single comment's thread
view the rest of the comments
[–] 17 points 4 months ago (2 children)

Tokens are an AI context of units of text. Could be a character or a word. Why use the term tokens instead of words? Who knows. Maybe because some units aren’t words? The tech industry delights in creating stupid terms for things that already had terms.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 4 points 4 months ago

    There are to many words to have each be it's own input, but bundling frequently used snippets is more efficient than using characters.
    Then the investors came and decided to measure success by wasted energy and the rest is history.

  • source
  • parent