Generative AI & LLMs beginner

Token

The unit a model actually reads and writes — usually a word fragment, not a word.

Models do not see letters or words; they see tokens, which are chunks of characters produced by a tokenizer. English averages roughly four characters per token, so 750 words is about 1,000 tokens. Tokens are also the billing unit and the unit of the context window, which makes this the most commercially relevant term in the glossary.

In practice: ‘unbelievable’ might split into ‘un’, ‘believ’, ‘able’ — three tokens, one word.