If you have used AI tools for a while, you have probably seen the words tokens and context window in pricing pages, usage limits or error messages. These two concepts explain many things people find puzzling about AI assistants.
Updated September 2026: we added links to OpenAI’s token guidance, Google’s long-context announcement and research on how models use long inputs.
What is a token?
Language models do not read text letter by letter or word by word. They break it into tokens: chunks that are often whole common words, parts of longer words, punctuation or spaces.
As a rough rule of thumb for English, one token is about four characters, or three-quarters of a word, so 1,000 tokens is roughly 750 words. Other languages, code and unusual words can use more tokens for the same amount of text.
Tokens matter because:
- Pricing for AI APIs is usually per token, counted separately for input and output.
- Usage limits on apps are often based on token volume behind the scenes.
- Speed: models generate one token at a time, so long answers take longer.
What is a context window?
The context window is the maximum amount of text a model can consider at once, measured in tokens. It includes everything: system instructions, your messages, the model’s replies, uploaded documents and search results.
Context windows have grown enormously. Early chatbots could handle a few thousand tokens; many current models accept hundreds of thousands, and some a million or more. When Google announced Gemini 1.5 Pro in February 2024, it said a 1-million-token window could hold more than 700,000 words or a codebase of over 30,000 lines. That is enough for long reports, several books or a sizeable codebase.
Analogy The context window is like a desk. Everything the model can see right now has to fit on it. Once the desk is full, older papers have to be removed or summarized to make room.
Why AI seems to forget
In a very long conversation, the earliest messages may eventually be dropped or summarized to fit the window. Even within the limit, models do not always pay equal attention to everything: details buried in the middle of a huge input can be missed. Researchers call this being "lost in the middle": models tend to use information at the start and end of a long input better than information in the middle. That is why an assistant may overlook an instruction you gave an hour ago.
Some assistants also have separate memory features that save facts between conversations. That is different from the context window and is usually something you can view and edit.
Practical tips
- Start a new chat for a new task. It keeps the context focused and often improves answers.
- Restate key instructions in long sessions.
- Summarize and carry forward. Ask for a summary of decisions so far, then paste it into a fresh chat.
- Ask targeted questions about long documents rather than relying on one giant summary, as covered in our document summarizing guide.
- Use projects or reference files for material you use repeatedly, as described in our custom assistant guide.
Why bigger is not everything
A huge context window is useful, but processing more tokens costs more and takes longer, and quality on very long inputs varies between models. Techniques such as retrieval, which pulls in only the relevant parts, remain valuable even with large windows.
Why this matters in practice
Tokens are how AI measures text; the context window is how much it can hold in mind at once. Knowing both helps you manage costs, work within limits and get more consistent answers from long conversations.
Sources
- What are tokens and how to count them?, OpenAI Help Center
- Introducing Gemini 1.5, Google, February 2024
- Lost in the Middle: How Language Models Use Long Contexts, arXiv, 2023
