AI Glossary
Token
A small unit of text processed by an AI model, often representing a word, part of a word, or punctuation.
Token
Overview
When people interact with AI systems, they typically think in terms of words and sentences.
AI models think differently.
Before text can be processed, it must first be broken into smaller units called tokens. Depending on the model, a token may represent an entire word, part of a word, punctuation, or another piece of text.
This process is known as Tokenization.
Tokens serve as the basic building blocks of language processing. Every prompt entered into a chatbot and every response generated by a model is ultimately processed as tokens.
A helpful way to think about tokens is building blocks. Just as individual blocks combine to create larger structures, tokens combine to form sentences, paragraphs, and conversations.
Tokens are also closely connected to the concept of a Context Window. The amount of information a model can process is typically measured in tokens rather than words.
Understanding tokens helps explain how language models process information and why context limits exist.
Why It Matters
Tokens form the foundation of how language models understand and generate text.
Real-World Example
A chatbot converts your prompt into tokens before generating a response.