Where your context window actually goes

The spec sheet says the context window is huge. Your agent still runs out of room by turn thirty, and it starts making worse decisions well before that. The fix is to treat the window like a budget: know the line items, find the leak, and put limits on it before it fills up.

The line items

Every call to the model sends the whole context again. In an agent loop, that context is made of six things:

• System prompt. Instructions, rules, persona. Usually small and fixed.
• Tool definitions. The name, description and parameter schema of every tool. Paid on every single call, whether the tool gets used or not.
• Reference docs. Retrieved chunks, a style guide, a codebase summary.
• History. Every user message and every model reply so far.
• Tool results. Every file read, search result, API response and command output so far.
• Output reserve. Room for the model to think and answer. If you do not leave it, the reply gets cut off or the request fails.

At turn one, most of the window is empty. By turn thirty, it is not, and the line item that grew is almost never the one people expect.

Measure before you cut

The chart above is illustrative. Yours will be different, so measure it. Log the size of each line item on every turn. Most model APIs return the input token count with each response, and most providers offer a token counting endpoint or tokenizer. If you just need a rough number, four characters per token is a fair estimate for English prose. Code and JSON usually run denser than that.

def rough_tokens(text: str) -> int:
    return len(text) // 4          # rough. use your provider's counter for real numbers


def log_budget(turn, system, tools, docs, history, tool_results):
    parts = {
        "system": rough_tokens(system),
        "tools": rough_tokens(json.dumps(tools)),
        "docs": sum(rough_tokens(d) for d in docs),
        "history": sum(rough_tokens(m["content"]) for m in history),
        "tool_results": sum(rough_tokens(r) for r in tool_results),
    }
    total = sum(parts.values())
    print(f"turn {turn:>3}  total {total:>7}  " +
          "  ".join(f"{k} {v / total:.0%}" for k, v in parts.items()))

Run your agent on a real task with this in the loop. The line that grows fastest is your leak. In most agents it is tool results.

The usual leak: tool results

A file read returns the whole file. A search returns fifty hits with full snippets. A failing test run prints four thousand lines of stack trace. The model needed maybe a tenth of it, and the rest stays in context for every call that follows.

Put a cap on every tool's output. Keep the head and the tail, because the head says what it is and the tail usually says what went wrong. Then tell the model what was cut and how to get more.

def cap(text: str, limit: int = 8_000, keep_tail: int = 2_000) -> str:
    if len(text) <= limit:
        return text
    head = text[: limit - keep_tail]
    tail = text[-keep_tail:]
    cut = len(text) - len(head) - len(tail)
    return (f"{head}\n\n[... {cut} characters cut. Call again with "
            f"offset/limit or a narrower query to see more ...]\n\n{tail}")

Better still, design tools that return less in the first place:

• Return references, not contents. A search tool can return file paths and line numbers, and let the model read only what it picks.
• Paginate. Give list and read tools offset and limit parameters with small defaults.
• Strip the noise. HTML pages become text. API responses drop fields the model never uses. Logs drop timestamps unless timing is the question.
• Clear stale results. Once the model has read a file and acted on it, an old copy twenty turns back is mostly dead weight. Replace it with a one-line note that it was read.

History: compact it like a shift handoff

Long conversations need compaction at some point. Keep the last several turns word for word, because that is where the current work is. Replace everything older with a running summary.

The quality of that summary decides whether the agent stays on track. Write the summarization prompt like a handoff note between shifts, not like a book report:

• Decisions already made, and why
• Open tasks and what is blocking them
• Exact file paths, IDs, URLs and names, copied verbatim
• Things that were tried and did not work
• What the user asked for, in their words

Drop greetings, restated plans and anything that has been superseded. Summaries lose detail every time, so trigger compaction at a threshold (say 60 to 70 percent full), not on every turn.

Tool definitions: only bring the tools for this job

Twenty tools with long descriptions and nested schemas can cost thousands of tokens on every call before anything happens. Load tools by task. A code review agent does not need the deploy tools. If you have a large catalog, give the model a small search tool that finds the right tool and loads its definition on demand.

Keep descriptions precise rather than long. One sentence on what the tool does, one on when to use it, and clear parameter names will do more than three paragraphs.

Order the prompt for caching

Most providers can cache the start of a prompt and reuse it on the next call, which is faster and cheaper. The cache generally matches on an exact prefix, so the order of the context matters. Put what stays the same at the top and what changes at the bottom.

The classic mistake is a line like Current time: 16:59:46 at the top of the system prompt. It changes every call, so nothing after it is ever reused. Move it to the latest user message. The same goes for per-request IDs, randomly ordered tool lists and anything else that varies at the top.

A starting budget

Every agent is different, but a written budget beats none. Here is a starting point for a general coding or research agent. Adjust it after you have measured.

Shift checklist

• Token count per line item logged on every turn
• Every tool's output capped, with a note saying what was cut
• Search and list tools return references and paginate
• Compaction at a threshold, written as a handoff note
• Tools loaded by task, not all at once
• Nothing that changes per call above the stable prefix
• Output reserve written down and never spent

Next on the shift: prompt injection, posted as safety signs.

Filed by the operations desk at The Paperclip Factory. We make workwear for people who ship models. If someone on your team keeps the evals green, the AI engineer gifts shelf is down the hall.