Skip to content

What Is Context Window? A Plain-English Definition

A context window is the ceiling on how much text, measured in tokens, a large language model (LLM) can consider at once when it handles one request, and that ceiling includes the answer it is writing. For AI agents in a business, it decides how much of the task, the history and the company's own knowledge can sit side by side in a single step.

How does a context window work?

Each request bundles everything the model should see into one input. The model reads all of it, then generates a reply that also takes up room in the window. When the request ends, nothing is retained inside the model itself.

Anthropic's documentation on context windows lists what counts: the system prompt, every message including tool results, images and documents, the tool definitions, and the generated output with any extended thinking. As of October 2026, current Claude models offer a 1M-token window, while some older ones are limited to 200k. A token is roughly a fragment of a word; the exact split depends on the tokenizer.

In an agent workflow the window fills up step by step. A support agent that looks up a customer record, opens a service manual and queries a ticketing system appends each of those tool use results to the history, and the whole history is sent again on the next call.

Context window vs memory vs RAG

The context window is working memory, not long-term memory and not training data. What a model learned in training is baked in; what sits in the window applies to this request only. Any "memory" that persists across sessions is a feature of the application around the model, which saves information and feeds it back in later.

A large window does not make retrieval-augmented generation (RAG) obsolete. A company's document estate, years of contracts, manuals and tickets, usually will not fit even into a million tokens. Full windows cost more and respond more slowly. Quality also suffers: Anthropic's documentation states that accuracy and recall degrade as token count grows, a pattern it calls context rot, and the research paper "Lost in the Middle" found that models often use information at the start or end of a long input better than material buried in the middle. RAG keeps the window small by selecting only the passages that matter.

What to watch out for

In practice, what goes into the window matters more than its maximum size. An agent handed the three relevant paragraphs of a procedure tends to perform more reliably than one handed the entire binder.

  • Measure usage: APIs report token consumption per request; Claude rejects inputs that exceed the window with an error.
  • Plan for long runs: drop or summarise old tool output before the window fills.
  • Position matters: put the task and key rules first, the concrete question last.
  • Caching is not compression: prompt caching lowers the price of repeated content, but it still occupies space.

To see how these limits play out in workflows such as quoting or field service, read the overview of AI agents for business. More terms are explained in the AI glossary.

Frequently asked questions

Is the context window the same as an AI assistant's memory?

No. It only covers the current request. Products that remember earlier sessions store that information separately and load it back into the window when needed.

Does a larger context window make answers more accurate?

Not automatically. Model providers and researchers report that recall drops as inputs grow, so a focused prompt often beats a full one.

Why do long agent runs get slower and more expensive?

Every step re-sends the growing history, and providers bill per token. Trimming or summarising old tool output keeps both cost and latency in check.

Sources

  1. Anthropic: Context windows (Claude documentation, accessed October 2026) platform.claude.com
  2. Liu et al.: Lost in the Middle – How Language Models Use Long Contexts arxiv.org