Tokenisation

The process of breaking text into smaller units, called tokens, that an AI model can process numerically. A token might be a whole word, part of a word, or a punctuation mark. Large language models read and generate text one token at a time, which is why they are billed and measured in tokens rather than words. (See also: Large language model, Context window)