Tokens and Context Windows: The Working Space of a Language Model

A language model does not read a conversation as pages, sentences, or human memories.

It receives a sequence of numbered pieces called tokens and predicts what token should come next.

A context window is the limited token workspace available for instructions, conversation, retrieved documents, tool results, and the model's answer.

Understanding that workspace explains why long chats can forget details, why document size affects cost, and why permanent memory requires an external system.

A concrete example: reviewing a long contract

Suppose an assistant must compare a 70-page contract with a company policy.

The request consumes context for:

• the system instructions,
• the user's question,
• conversation history,
• the contract,
• the policy,
• formatting examples,
• and the generated comparison.

If those inputs approach the model's limit, the application must remove, summarize, or retrieve only selected material. It must also reserve enough tokens for the answer.

What a token is

A token is a chunk produced by a tokenizer. It may be:

• a common whole word,
• part of a long word,
• punctuation,
• whitespace,
• a number fragment,
• or a piece of code.

The exact breakdown depends on the model's tokenizer. “Technology” might be one token for one model and several for another. Languages, unusual names, and dense code can use tokens at different rates.

Tokens are therefore not identical to characters or words.

Why models use tokenization

A model needs a finite vocabulary of units it can represent numerically.

Character-only vocabularies produce very long sequences. Whole-word vocabularies struggle with new words, spelling variations, and enormous dictionaries. Subword tokenization balances those concerns: common patterns can be compact while unfamiliar text can still be assembled from smaller pieces.

Input and output share capacity

Applications often describe a model as having a context window of a certain size. That capacity usually covers both the supplied input and generated output.

If input occupies nearly all available space, little remains for a detailed answer. A system should set output limits intentionally and reject or reduce oversized requests before inference.

Context is not permanent memory

The context window is temporary working input for one model call. When the call ends, the model does not automatically retain it for the next conversation.

An application can create the appearance of memory by storing:

• previous messages,
• user preferences,
• summaries,
• documents,
• or structured facts

and inserting relevant items into a later context. The storage and retrieval system provides persistence; the model consumes what it is shown.

Why long conversations lose details

A chat application cannot keep appending forever. Once history exceeds its budget, it may:

• remove old turns,
• summarize them,
• retain selected facts,
• or retrieve relevant past messages.

Each strategy can lose nuance. A summary may omit an exception, while similarity search may fail to retrieve an important but differently worded decision.

Position affects attention

Even when text technically fits, a model may not use every detail equally well. Important evidence buried inside a very long context can receive less effective attention than clear instructions and focused evidence.

Place critical rules prominently, separate documents clearly, and ask the model to cite the exact evidence it used. Larger capacity does not remove the need for information design.

Context budgeting

Treat context like a budget with named allocations:

• fixed system and safety instructions,
• recent conversation,
• persistent user facts,
• retrieved evidence,
• tool outputs,
• examples,
• and answer allowance.

Measure actual token counts with the model's tokenizer when precision matters. Character estimates are useful for early checks but can be wrong for multilingual text or code.

Chunking long material

When a document is too large, applications divide it into chunks.

Chunk boundaries should preserve meaning. A clause and its exception belong together; a table header should accompany its rows. Overly small chunks lose context, while oversized chunks consume capacity and dilute retrieval.

Useful systems keep metadata such as document, section, page, date, and permissions with each chunk.

Summarization as compression

Summaries can compress old conversation or large documents, but compression is lossy.

Use structured summaries for facts that must survive:

• decisions,
• constraints,
• unresolved questions,
• definitions,
• and source references.

For high-stakes work, preserve links to the original material so the system can retrieve exact wording instead of trusting a summary alone.

Prompt injection consumes the same channel

Retrieved documents and web pages enter the context as tokens too. They may contain text pretending to be instructions.

Applications must distinguish trusted instructions from untrusted content, restrict available tools, and avoid treating every sentence in context as equally authoritative. Token boundaries do not create security boundaries.

Cost and latency

Processing more tokens generally increases computation, cost, and response time. Long prompts also make caching and debugging harder.

Do not fill a large window merely because it exists. Retrieve relevant evidence, remove repeated boilerplate, and use structured state where exact fields are better than prose history.

Common misunderstanding

A model with a 100,000-token window has not “learned” the 100,000 tokens supplied in a request. It can use them temporarily while producing the response.

Changing model weights through training is different from adding text to context. Context supplies current working information; training changes learned behavior and patterns.

Designing a reliable conversation

For a durable assistant:

1. keep authoritative instructions separate,
2. store persistent facts outside the model,
3. retrieve only relevant history,
4. label sources and timestamps,
5. reserve answer capacity,
6. test conversations near the limit,
7. and let users inspect or correct remembered facts.

This turns “memory” into an explicit product capability rather than an accidental side effect of replaying chat logs.

Knowledge check

1. Why is a token not the same as a word?
2. What information consumes a context window?
3. Why must an application reserve output capacity?
4. How does persistent memory differ from context?
5. Why can a larger context still miss an important detail?

The one idea to remember

Language models process temporary sequences of tokens inside a finite context window. Reliable applications budget that space, retrieve focused evidence, preserve important state externally, reserve room for output, and never confuse a long context with permanent memory.