If you have used AI tools for a while, you have probably seen the words tokens and context window in pricing pages, usage limits or error messages. These two concepts explain many things people find puzzling about AI assistants.
What is a token?
Language models do not read text letter by letter or word by word. They break it into tokens: chunks that are often whole common words, parts of longer words, punctuation or spaces.
As a rough rule of thumb for English, one token is about four characters, or three-quarters of a word, so 1,000 tokens is roughly 750 words. Other languages, code and unusual words can use more tokens for the same amount of text.
Tokens matter for three practical reasons. AI APIs usually charge per token, counting input and output separately. Usage limits in apps are often based on token volume behind the scenes. And models generate one token at a time, so long answers take longer.
What is a context window?
The context window is the maximum amount of text a model can consider at once, measured in tokens. It includes everything: system instructions, your messages, the model’s replies, uploaded documents and search results.
Context windows have grown enormously. Early chatbots could handle a few thousand tokens; many current models accept hundreds of thousands, and some a million or more. When Google announced Gemini 1.5 Pro in February 2024, it said a 1-million-token window could hold more than 700,000 words or a codebase of over 30,000 lines. That is enough for long reports, several books or a sizeable codebase.
Analogy The context window is like a desk. Everything the model can see right now has to fit on it. Once the desk is full, older papers have to be removed or summarized to make room.
Why AI seems to forget
In a very long conversation, the earliest messages may eventually be dropped or summarized to fit the window. Even within the limit, models do not always pay equal attention to everything: details buried in the middle of a huge input can be missed. Researchers call this being "lost in the middle": models tend to use information at the start and end of a long input better than information in the middle. That is why an assistant may overlook an instruction you gave an hour ago.
Some assistants also have separate memory features that save facts between conversations. That is different from the context window and is usually something you can view and edit.
Practical tips
Start a new chat when you start a new task. It keeps the context focused and often improves answers. In long sessions, restate your key instructions now and then. If a conversation has run long, ask for a summary of the decisions so far and paste it into a fresh chat.
With long documents, ask targeted questions rather than relying on one giant summary, as covered in our document summarizing guide. For material you use over and over, set up a project or reference files, as described in our custom assistant guide.
Why bigger is not everything
A huge context window is useful, but processing more tokens costs more and takes longer, and quality on very long inputs varies between models. Techniques such as retrieval, which pulls in only the relevant parts, remain valuable even with large windows.
Why this matters in practice
Tokens are how AI measures text; the context window is how much it can hold in mind at once. Knowing both helps you manage costs, work within limits and get more consistent answers from long conversations.
Sources
- What are tokens and how to count them?, OpenAI Help Center
- Introducing Gemini 1.5, Google, February 2024
- Lost in the Middle: How Language Models Use Long Contexts, arXiv, 2023
Related reading: machine learning basics in plain English.
