Context Window Management in Long-Running Agents

When a long-running agent starts making mistakes, the instinct is to reach for a bigger window. More tokens, more room, problem solved, or so the thinking goes. But that instinct misdiagnoses what's actually happening. Per Zylos AI research, context degradation sets in at roughly half of a model's nominal window capacity, long before the window is anywhere near full. The cause is the attention mechanism itself losing resolution as sequences stretch out. It's that the attention mechanism itself loses resolution as sequences stretch out, which means the model's ability to weigh and recall information gets worse well before it runs out of room to store it.

Manage context by controlling what enters the window, what stays, and what gets cut, not by buying a bigger window. Degradation starts at roughly half of a model's nominal capacity, and information in the middle of a long context loses 30% or more accuracy. Five strategies dominate: sliding windows, recursive summarization, structured state (scratchpads), retrieval-based ephemeral context, and dynamic context routing. Controlled research found that simple observation masking, which swaps older tool outputs for a placeholder, cut cost by 50% while matching LLM summarization on solve rate, and a hybrid of the two cut cost further. Put static instructions at the top of the prompt so caching works, and keep durable memory outside the window across sessions.

The evidence for this is specific. Information in the middle of a long context sees accuracy drops of 30% or more. It sits there because of where it sits. That's a structural problem, not a headroom problem, and no amount of window expansion fixes a structural issue. Providers have spent the last few years racing to ship larger and larger windows, and that race has genuine value for some use cases. But for long-running agents wrestling with reasoning quality, it treats the wrong symptom.

Why does this matter enough to reorganize how teams build agents? Cost is the sharper edge of the same problem. Per GetMaxim research, a long conversation costs orders of magnitude more than a short one on the same model. So a bigger window doesn't just fail to fix reasoning quality, it scales the cost of carrying bad context along with it. Every stale instruction, every irrelevant tool output, every contradictory note sitting in that window gets paid for again, every single turn. The discipline that's actually needed isn't window expansion. It's deliberate control over what enters the window, what's allowed to stay, and what gets cut loose.

What "context rot" looks like in a long session

Context rot is the term for what accumulates when a session runs long: stale content, contradictory content, low-signal content, all piling up together. It doesn't announce itself. A session can start lean and small, then swell fast, going from a few thousand tokens to tens of thousands within a handful of exchanges if nothing is actively managing it.

Four failure patterns recur constantly in production systems. The first is front-loading: stuffing context in "just in case" it's needed later, which bloats the token count immediately and dilutes the signal from turn one. The second is an ordering mistake, placing the user's query ahead of the static system instructions. That seems harmless, but it breaks provider-level prompt caching for everything that follows, and it resurfaces later in a much bigger way. The third is a delegation problem: passing an entire orchestrator's context down to a sub-agent, which means every layer of delegation compounds the irrelevance instead of trimming it. The fourth is quieter and arguably the most dangerous. A hard rule gets set early in a session, something like "always confirm before deleting," and that rule works fine at first. But as the session runs on, the rule drifts toward the middle of the context window, exactly where accuracy measurably drops, and the rule erodes without anyone noticing it went.

For enterprise agents specifically, context rot takes on a sharper, more structural shape. Call it semantic conflict. A data dictionary defines a field one way, but the schema has since changed and nobody updated the entry. Two departments define the same term differently, one calling "active user" someone who logged in this week, another meaning someone with a paid subscription. Lineage gaps mean the agent can't trace where a column's data actually came from, so it has no way to verify what the column means in the first place. None of that is a "too many tokens" problem. It's a governance problem wearing a context management costume.

The scale of the damage is striking. Per Zylos AI's research, roughly 65% of enterprise AI failures trace back to context drift and memory loss during multi-step reasoning, not to any raw limitation in the model itself. That's a striking number, because it means the bottleneck is what the model is being fed, and when, and in what shape, not model capability. It's what the model is being fed, and when, and in what shape. Which is, in a strange way, good news: a predictable failure is a manageable one. If the rot follows patterns, then the patterns can be engineered against.

How should I manage context in AI agents that run for a long time?

Five approaches dominate how practitioners actually fight context rot today, and each one trades something away to get something back. Per MachineLearningMastery and GetMaxim, those five are sliding windows, recursive summarization, structured state management, ephemeral context through retrieval, and dynamic context routing. None of them is the obvious right answer. The right one depends on what the agent is doing, how long the session runs, and how much the team is willing to spend.

Sliding windows are the simplest option on the list. Drop the oldest messages, keep the most recent N turns, anchor everything to a system prompt that never moves. It's the cheapest and fastest, since it requires no additional AI processing. The cost appears later, in what might be called digital amnesia: if the agent already solved a problem earlier in the session and encounters something similar again, it has no memory of the earlier fix. That risks loops, the agent re-deriving (or re-failing) the same thing twice.

Recursive summarization tries to fix that gap by compressing older messages into a running summary instead of just discarding them. That preserves the mission, the overall plot of a long session, even after the raw detail is gone. But summarization behaves like lossy image compression: the broad shape survives, the fine grain doesn't. The agent remembers that something happened, without remembering exactly what.

Structured state management, often called a scratchpad approach, works differently again. Instead of letting a chat transcript pile up turn after turn, the agent maintains a structured document, JSON or Markdown, that tracks goals, completed steps, key findings, and open questions. Raw tool output gets compressed or thrown away; only its distilled essence makes it into the scratchpad. This is emerging as the preferred pattern for multi-step reasoning tasks, and it's remarkably token-efficient, since scratchpad updates replace the need to carry the entire conversation forward. Its weakness is schema rigidity. Whatever the developer didn't anticipate falls outside the schema's boundaries, and gets silently dropped, even if it turns out to matter.

Ephemeral context through retrieval-augmented generation (RAG) takes yet another approach. Push the full history out to an external vector database, and at each turn pull back only the subset that's currently relevant, so the context window only ever holds the current working set. It's arguably the most sophisticated of the five, well-suited to long-horizon agents that need to write down and later retrieve findings across a sprawling trajectory. But it lives and dies by retrieval quality. Content that's poorly indexed or embedded may simply never surface again, a blind spot that becomes visible only at the exact moment it costs the agent something important.

Dynamic context routing flips the order of operations. Instead of pruning after the fact, classify the intent of a query first, then retrieve and inject only the specific context that matches it, so the window stays signal-dense from the very first token. Atlan's context routing layer is a working example: a query about a specific field gets mapped to that field's glossary entry, its schema definition, and its lineage record, rather than dumping the entire data dictionary into the window. The cost here is maintenance. The routing layer has to be kept current as schemas and ontologies evolve, and a routing layer that's gone stale is its own form of context rot, just one layer removed.

None of these five wins outright. A team running short customer-support sessions might do fine with a sliding window.

What the evidence says about summarization versus simpler baselines

Here's where the intuition gets challenged directly. It would be reasonable to assume that LLM-based summarization, being the more sophisticated of the five strategies, wins on performance even if it costs more. Controlled research says otherwise.

Observation masking doesn't try to be clever. Older tool outputs just get replaced with a placeholder, something like "Previous 8 lines omitted for brevity," while the most recent observations stay intact and in full. No model call, no compression logic, just a blunt cut.

The result should give pause to anyone defaulting to summarization by habit. Observation masking cut cost in half relative to a raw, unmanaged agent, a full 50% reduction, while matching or slightly beating the task solve rate that LLM summarization achieved. The simpler, cheaper method held its own against the fancier one, and sometimes edged it out. Across the trials, the LLM-Summary strategy wasn't able to consistently or significantly outperform the simpler Observation Masking baseline.

Two mechanisms explain why. First, summarization isn't free: the summarization calls themselves eat into the total inference budget, sometimes a meaningful chunk of it, for the stronger models tested. Second, and more interesting, summaries can hide the exact signal an agent needs most: failure. An agent fed a smoothed-over, coherent-sounding account of its own prior steps may not register that it's repeating a dead end. The summary papers over the mess, and the mess was informative.

None of this writes summarization off entirely. The same study found that a hybrid combination of LLM-Summary and Observation Masking achieves further cost reductions compared to either approach alone, suggesting neither is optimal in isolation. A hybrid combination produces the actual gains. Benchmark the boring baseline before building the elaborate pipeline. The added complexity of a summarization layer needs to earn its keep, and the evidence says it often doesn't, not on its own. A JetBrains/TUM study (arXiv:2508.21433, presented at NeurIPS DL4Code 2025) tested LLM-based summarization against a simpler observation masking baseline.

How prompt caching changes the economics of every strategy

When the static portion of the prompt (system message, common instructions, frequently retrieved chunks) is cached, the effective per-call cost drops dramatically, and most major providers support prompt caching as of 2026. Most major model providers support this by 2026. That's not a minor efficiency trick. It reframes the whole optimization approach. The goal shifts from "make the context as small as possible" to "make the context as cache-friendly as possible," because those two goals don't always point the same direction.

Cache hits depend entirely on structure. Static, high-reuse material needs to sit at the top of the prompt, with anything dynamic pushed below it, or the caching mechanism has nothing stable to latch onto. This is exactly why the ordering mistake flagged earlier, putting the user's query ahead of the system instructions, does more damage than it looks like on the surface. It doesn't just read a little sloppily. It silently kills cache efficiency for every downstream call in that session.

Sliding windows separate the locked system prompt at the top, which rarely changes, from the sliding history that follows it, and keeping those two cleanly separated pays off. Structured scratchpads cache well on the system prompt and schema definition, since those stay fixed, while only the scratchpad's contents shift turn to turn. RAG setups benefit when frequently retrieved chunks, the ones that show up across many different queries, get treated as effectively static once fetched, which can cut per-call cost substantially.

The upshot is that context management was never only a question of what enters the window. It's also a question of how that content is arranged inside it. Prompt ordering, once treated as a style preference or a matter of taste, is now a genuine engineering decision with real cost consequences attached to it.

Handling multimodal histories when visual evidence must survive compression

Everything above assumes the history being compressed is text. That assumption breaks for a growing class of agents: ones doing GUI automation, running image-search trajectories, or analyzing data that lives in charts and tables. For those agents, text-based compression strategies quietly throw away evidence that only exists in visual form.

Research from SenseTime, the National University of Singapore, and ShanghaiTech University (arXiv:2608.29897) treats this as a formal problem: context management under a token budget, framed as a transformation applied to history, with Visual Rendering evaluated as one representational strategy against four baselines, No Compression, Discard-All, Sliding Window, and Summarization. An image-search trajectory carries retrieved images and localized crops as its actual evidence, and a data-analysis trajectory carries tables, charts, document pages, whose spatial layout is exactly what gets lost when it's flattened down into prose. Compress that visual structure into a paragraph of text, and the paragraph can describe the chart, but it can't reproduce what the chart's layout was actually communicating.

The paper's proposed answer is VERA, short for Visual Evidence-Retaining strategy for long-horizon Agents. It's training-free and needs no supervised fine-tuning, no self-distillation, no reinforcement learning to work. VERA renders textual history visually when the task is text-centric, but for multimodal tasks, it keeps the native visual observations as-is rather than translating them into a text description. Across nearly every benchmark tested, that approach cut cumulative non-cache tokens by up to 63.1% compared to no compression at all. On the multimodal tasks specifically, where the visual evidence actually mattered, VERA came out on top for accuracy against every baseline tested.

The lesson generalizes past this one paper. The right compression strategy isn't fixed, it depends on what kind of evidence is being compressed. Teams building agents that operate inside visual environments need to treat representation itself, not just retention, as a first-class decision in how context gets managed.

Persistent memory across session boundaries, what survives when the run ends

Everything covered so far happens inside a single run. But what happens when the run ends? For an agent meant to persist across days or weeks, that's the harder challenge, and in-session strategies don't answer it.

The graph's state object exists only for the duration of one run; once the run ends, the state ends with it. The common workaround is to dump conversation history into a database or a vector store and manually pull it back into the next prompt. That works, technically, but it undercuts the whole point of in-session compression by stuffing prior context right back into the window on the next run.

Mem0 has emerged as a more targeted answer to this specific gap: a system built to store, retrieve, and update durable facts about a user, while the orchestration graph stays responsible for coordinating what happens inside a single session. That division of labor means personalization can carry across runs without dragging an entire prior conversation back into the window every time.

Multi-scope memory is the design pattern that actually makes this tractable. Every memory write gets tagged: a user_id scope for facts that should persist across sessions, an agent_id scope for facts tied to one specific agent instance, a run_id scope for facts that belong only to a single conversation, and an org_id scope for context shared across an organization. At retrieval time, those scopes get composed and ranked automatically, so the agent pulls in what's relevant to the current scope without pulling in everything that's ever been written.

Two problems remain open. First, cross-session identity resolution: when it's unclear whether two interactions came from the same person, identity becomes ambiguous, and memory attribution gets unreliable fast. Second, memory staleness, a subtler trap. A fact about a user's employer, retrieved constantly because it's useful, can quietly become wrong the moment that person changes jobs. High retrieval frequency doesn't signal continued accuracy, it's arguably the opposite: the more a stale fact gets used, the more confidently wrong the agent becomes. That's a harder problem than a memory nobody bothers to retrieve anymore, which at least fades quietly instead of doing active damage.

The infrastructure layer that makes context management operational at scale

Zoom out from any single strategy and a pattern becomes obvious: none of this works as a one-off script bolted onto an agent. Sliding windows, recursive summarization, structured state management, ephemeral context via RAG, and dynamic context routing are the five strategies. An agent that silently loses a hard rule to context rot, or serves a stale memory with total confidence, isn't failing loudly enough to get caught by a human watching a dashboard casually.

That's precisely why context management is drawing serious attention at the systems level, not just the application level. It's the subject of dedicated venues like AgenticOS 2026, a workshop built around a direct claim: the operating system abstractions built for processes, threads, files, and sockets were never designed for workloads this dynamic, this semantically loaded, this adaptive. Building an operating system layer that understands agent behavior at that level, rather than treating an agent like any other process, is still an open design question, not a solved one.

What's clear already is the shape of the problem. Context management isn't a feature to bolt onto an agent after the fact. It's closer to a discipline, one that spans how prompts get ordered, how memory gets scoped, how visual evidence gets represented, and how failure signals get preserved instead of smoothed away. Get that discipline right, and the agent reasons well for as long as it needs to run. Get it wrong, and no window size, however large, will save it.