What Is Token Limit in LLMs?

What Is Token Limit in LLMs?

Every so often an AI "forgets" the start of a long conversation, or refuses a giant document. Behind that is a hard constraint: the token limit. Models can only hold so much text in mind at once, and understanding that limit explains a lot of otherwise baffling AI behaviour. It's one of the most practical things to grasp when building with LLMs. Here's what a token limit is, in plain terms.

The short version

A token limit (or context window) is the maximum amount of text – measured in tokens – that a large language model can process in a single request, including both the input you send and the output it generates. A token is a chunk of text, roughly a word or part of one. Once you exceed the limit, the model can't consider the excess, which shapes how you design AI applications.

What a token is

Models don't read words exactly as we do; they break text into tokens – small chunks that are often a word, part of a word, or punctuation. As a rough guide, a token is about three-quarters of a word in English. Everything counts: your prompt, any context or documents you include, and the model's response all consume tokens from the same budget.

Why the limit matters

• It caps how much context you can give the model at once.

• Input and output share the same budget.

• Exceeding it means the model can't see the extra text.

• It affects cost, since you often pay per token.

• It shapes how you design prompts and long conversations.

How it shapes applications

The token limit is why long chats can lose earlier detail, why you can't just paste an entire library into a prompt, and why techniques like retrieval exist – to fetch only the relevant slices of a large knowledge base rather than sending all of it. Context windows have grown large in modern models, but they're still finite, so managing what you include remains a core design skill.

Working within it

Good AI engineering respects the token budget: summarising or trimming history, retrieving only relevant context, and structuring prompts efficiently. Retrieval-augmented generation exists largely to sidestep the limit by fetching just what's needed. Managing tokens well also controls cost. Our development team designs AI systems that use the context window efficiently, so features stay accurate and affordable even over large amounts of information.

FAQ

What exactly is a token?

A token is a chunk of text the model processes – often a word, part of a word, or punctuation. In English, a token averages roughly three-quarters of a word. Both your input and the model's output are measured in tokens.

Why does the AI forget earlier parts of a long conversation?

Because the conversation history counts toward the token limit. Once it grows past the context window, the oldest parts fall outside what the model can see, so it effectively forgets them unless they're summarised or reintroduced.

Do bigger context windows solve the problem?

They help a lot, letting models handle far more text at once, but they're still finite and larger contexts cost more. Managing what you include – through retrieval and summarisation – remains important even with large windows.

Sources

• Anthropic – Context Windows: https://docs.claude.com/en/docs/build-with-claude/context-windows

• OpenAI – Tokenizer & Docs: https://platform.openai.com/docs

• Anthropic Documentation: https://docs.claude.com/