Generative AI & LLMs intermediate

Tokenization

Cutting text into the tokens a model can process — and a quiet source of odd model behaviour.

A tokenizer maps text to integers using a fixed vocabulary learned from data. It explains several famous quirks: models miscount letters because they never see letters, and non-English text often costs more tokens for the same meaning because vocabularies skew English.

In practice: The same sentence in Greek can cost two to three times the tokens it costs in English.

Where this comes up