Glossary
Token (AI)
A small chunk of text, such as a word or part of a word, that a language model reads or writes.
Language models do not read whole words directly. They split text into tokens, which can be a short word, part of a longer word, a number, or a punctuation mark. Images and audio can be converted into tokens too. In English, a token is on average a bit shorter than a word.
Tokens matter in news because they are the unit for pricing and limits. AI companies usually charge developers per million input tokens (what is sent to the model) and per million output tokens (what it writes), with output usually priced higher. Context windows are also measured in tokens.
Token counts are not directly comparable across companies, because each model may split text differently. The same document can be a different number of tokens for two models.