Organizing Context in Multi-Agent Harnesses

Organizing Context in Multi-Agent Harnesses.

Why context management is the real bottleneck in multi-agent systems

Agents today are being asked to finish jobs. That's a different mode entirely: a closed loop that runs over many steps, not a single reply and done. The data shows the strain in agentic benchmarks rather than static ones. Static benchmarks like MMLU and HumanEval are running out of room at the frontier, but agentic benchmarks, SWE-bench, WebArena, OSWorld, TheAgentCompany, Terminal-Bench, still have plenty of headroom left, and simply scaling up the model doesn't close that gap.

So what does close it? Not a bigger model. What separates an agent that actually finishes a task from one that stalls out halfway through is the execution harness wrapped around the model, and inside that harness, the piece doing the heaviest lifting is context management. Guo et al. treat harness design itself as the real performance lever: success, efficiency, safety, how well an agent generalizes, all of it comes out of the interaction between model capability, runtime infrastructure, task structure, and how the work gets evaluated.

That question sits at the center of what follows here. In a multi-agent system, context is a distributed resource that has to be organized on purpose, spread across agents, sessions, and time. It's a distributed resource, spread across agents, sessions, and time, and it has to be organized on purpose. Get that organization wrong, and no amount of model horsepower saves the run.

What an agent harness does, with context at its center

Strip away the marketing language and a harness is just the runtime infrastructure wrapped around a language model, the thing that turns a stateless text-completion engine into something that can hold a goal and work toward it over multiple steps. The tools are purpose-built plumbing, designed as part of the system rather than bolted onto a model after the fact. It's purpose-built plumbing.

The harness draws a clean line: reasoning belongs entirely to the LLM. Everything else, dispatching tool calls, managing input and output, keeping the agent sandboxed so it can't do damage, routing messages between agents and users and outside services, logging every step for someone to debug later, that's the harness's job. It also owns memory across three different registers: what's sitting in the context window right now, a working scratchpad for the current session, and long-term storage in a vector store or database.

Guo et al. break the harness down into six coupled responsibilities that run the whole show: observation, context, control, action, state, and verification or governance. Context is just one piece of six. But it touches every other piece, which affects how failures in one area ripple through the rest of the harness.

A badly designed harness can flatten the output of even the best model available, while a well-built one can make a modest model punch well above its weight. A poorly designed harness can nullify the capabilities of even the most powerful LLM, while a well-designed one can amplify what a modest model achieves (context management is the most leveraged single responsibility).

How the context window became a contested resource across agents

Picture the context window as the scarcest shared resource in the whole system, because that's what it is. Memory retrieval, skill instructions, protocol schemas, tool descriptions, the model's own reasoning trace, they're all fighting over the same token budget, all at once.

Frontier models in 2026 ship with enormous windows, some configurations pushing past a million tokens. That sounds like the problem should be solved. It isn't. A bigger window doesn't remove the allocation problem, it just relocates it somewhere less obvious. More room to work with just means more competing demands filling that room.

Multi-agent systems make the relocation worse. Each agent carries its own context window, its own local slice of the world. Task state that needs to move between agents has to be explicitly routed, copied, or fetched, it does not propagate on its own. Every coordination message, every tool output, every sub-agent's result eats into the orchestrator's budget the moment it crosses that boundary. And when several agents run in parallel, nothing stops them from ending up with contradictory pictures of the same task.

No single module fixes this by itself. The right allocation depends on what phase of execution the task is in at that moment, which makes budget allocation a harness-level coordination problem, not a local one. Context management sits at the level of harness design, carrying the same weight as any other infrastructure decision, and that's a frame worth holding onto for the rest of this piece. It's infrastructure, with the same design demands as any other distributed resource, replication, consistency, and failure handling included.

The three memory layers agents draw on

Across the research, three tiers appear repeatedly, and each earns its place for a different reason.

Short-term, in-context memory is the active window itself: fast, free of latency, but gone the moment the session ends or the window fills up. Working memory, the scratchpad, holds intermediate task state written and read during a session, sometimes structured as JSON or markdown, sometimes loose text, and it survives individual model calls but not the session as a whole. Long-term memory lives in vector stores, databases, episodic logs, and it survives across sessions, but it costs you: explicit retrieval, higher latency, no free lunch.

The AIOS framing treats this as a kernel's job: an LLM-agent operating system isolates memory, storage, context, scheduling, and access control so it can manage several concurrent agents at once, and the three-tier structure above is exactly what that kernel is managing. Separately, the Deakin University survey lays out five capabilities agents need, sense, learn, reason, predict, act, and argues context awareness is what lets an agent adapt those five to a changing environment; the memory tiers are the mechanism by which context survives across time.

Three tensions produce the friction in all of this, driven by competing design constraints that don't resolve cleanly. Speed against capacity: in-context is quick but small, long-term is roomy but slow. Freshness against persistence: working memory changes constantly, long-term memory needs deliberate rules for when to update it. Local against shared: each agent only ever reads its own window, so long-term storage becomes the natural place where sharing actually happens.

One more point that shouldn't get buried: an agent with no access to persistent memory across sessions is, per the agent OS literature, just an expensive chatbot. Persistence is the entry fee for doing real work. It's the entry fee for doing real work.

Four strategies the harness uses to manage what enters the context window

The context manager has one job, really: decide what gets into the next model call, and in what shape. It selects, it compresses, it reorders, it refreshes, all before the next step fires.

Four strategies recur repeatedly. Summarization compresses older turns and past execution history into something shorter, keeping the decisions that matter while freeing up tokens for the step in front of the agent right now. Priority-based eviction removes or demotes entries whose relevance has faded, which only works if there's a real signal behind it, recency, a task-phase tag, semantic similarity to what's active now. Staged loading holds detailed procedural guidance back until the harness detects a matching task pattern, rather than dumping it all in at session start, so budget doesn't go to waste on instructions nobody's using yet.

None of these four run in isolation. The harness has to orchestrate them together, because what's optimal shifts with the phase of execution: early planning might call for more memory and less procedural detail, late execution often wants the opposite. The OptiMAS paper backs this up directly, introducing a dual-track memory mechanism that keeps performance climbing over long optimization runs, which is strong evidence that memory design is itself something you optimize, not just a setting you configure once and forget.

How shared state flows, or fails to flow, across agent boundaries

Context management, in a lot of these systems, is really just the tax paid for not having a formal shared substrate. Systems build increasingly clever workarounds precisely because the underlying problem, agents needing a common view of the world, was never solved directly.

A few concrete examples show how different teams have approached the same wall from different angles. L2MAC's Control Unit manages the model's context window alongside an external memory file store, which lets it produce codebases far too large to fit inside any single context window. MetaGPT runs a publish-subscribe pool: agents publish what they produce, others subscribe to the slice they actually need. SoA scales its agent pool dynamically, adding agents to spread the context load rather than cramming more into one window. Cogito separates memory into three tiers, ephemeral, session, persistent, specifically to cut down on context leaking across agent boundaries.

Phillips et al. identify five coordination patterns, Generator-Verifier, Orchestrator-Subagent, Agent Teams, Message Bus, Shared State, and land on Orchestrator-Subagent as the default recommendation. Each pattern answers a different way to the same question: who actually owns the authoritative version of context?

Skip that question, and three failure modes occur reliably. Context drift, where agents act on task-state views that have quietly gone stale. Redundant retrieval, where multiple agents fetch the same artifact independently, burning budget and risking version skew between copies. And silent contradiction, maybe the nastiest one: two sub-agents produce results built on incompatible premises, both land on the orchestrator's desk, and nothing in the system flags that they disagree.

What would actually fix this? A harness substrate that gives every agent a principled, queryable view of task state, something they can check against on demand rather than guess at. That's the direction the research points toward. It remains an open problem. It's an open problem.

How multi-agent topology shapes context requirements before a line of code is written

Before anyone writes a single function, the topology chosen already decides how hard the context problem will be. Five patterns recur across the research, and each one carries its own context burden.

Single-agent ReAct keeps everything in one window, one agent, no boundary to cross, so context management stays entirely local, which is why it's the right fit for simple tool-use jobs. Plan-and-Execute splits planning from execution: a planner produces a structured plan, executor agents consume it, and the real discipline lives in that plan artifact itself, how much detail it carries, what format it's in, whether it stays synced as things change. Hierarchical supervisor setups put a supervisor in charge of routing to specialized workers; the supervisor needs enough task state to route well without drowning in it, while workers only need their own slice. Maker-checker requires both the actor and the verifier to see the same artifact and the same provenance behind it, so shared context becomes the artifact plus the criteria used to judge it, and keeping that in sync matters enormously. Network or swarm topologies let peer agents share a scratchpad and talk freely, which works fine until nobody enforces write discipline, at which point the shared scratchpad turns into noise.

Phillips et al. land on Orchestrator-Subagent as the sensible starting default, because it hands clear ownership of the authoritative context to one place and limits what sub-agents even need to look at. Most production teams do better starting single-agent, finding where that pattern caps out on quality, and only then adding topology. Add coordination too early, and the overhead of managing multiple agents swallows whatever quality gain topology was supposed to buy.

Plan-and-Execute is also where cost and context considerations start to line up. Run a capable model for planning, hand execution off to cheaper models, and costs drop substantially, but only because the plan artifact carries enough structure to make that handoff work. The plan is the context handoff. Without it, the cheaper execution model has nothing solid to work from.

Graduated context management: moving from soft limits to system overrides

The naive move, when the window fills up, is to just cut the oldest content and keep going. Cheap to build. Often disastrous in practice, because it throws away state the agent still needs to finish the job.

OptiMAS offers a better structure: a three-tier capacity envelope with graduated interventions. In the proactive phase, the agent gets soft-limit warnings and skill protocols that nudge it to break its own workflow into phases, each one starting with a clean slate and a distilled summary of what came before. If that's not enough, the intermediate phase kicks in: the framework starts compressing and evicting lower-priority content on its own, guided by phase signals coming from the agent's own reasoning. Only if none of that works does the override phase activate, system-level controls stepping in because the agent failed to manage itself. That's the last resort.

What's notable about the proactive phase is what it does to the relationship between agent and framework. Context management changes from something the system does to the agent into something the two do together, a real collaborative protocol, because the agent becomes an active participant in managing its own memory rather than just a passive consumer of whatever the framework hands it.

Longer-horizon performance, it turns out, has less to do with cramming more into the prompt and more to do with keeping task state coherent as time goes on. That's the whole argument for graduated management over just buying a bigger window. But none of it works without a signal for phase, task decomposition, planning artifacts, explicit markers, something the context manager can actually use to decide what gets compressed, evicted, or loaded next.

What observability over context looks like across teams

They have the tools. What they don't have is a visible map of what's deciding what, or a record of what actually happened during a run. That gap sits in orchestration and observability, not in model capability.

Real context observability means a few concrete things, not vague dashboards. Logging what entered the context window on every single call, in what order, at what token cost. Tracing every retrieval, what got pulled from long-term memory, when, and whether the agent even used it once it arrived. Phase markers, so an engineer can reconstruct exactly what the agent's context looked like at any point deep into a long-running task. And trace correlation across agents, so when an orchestrator hands context to a sub-agent, that sub-agent's execution trace links straight back to its parent instead of floating free.

A handful of vendors are splitting up this layer right now: Langfuse, Helicone, LangSmith, Arize Phoenix, Braintrust, alongside OpenTelemetry-native stacks like Honeycomb, Datadog, and Grafana Tempo. OpenTelemetry's GenAI semantic conventions are still sitting in Development, experimental status as of mid-2026. That's not nothing, it gives teams a vendor-neutral baseline to instrument against, but it's also a signal that the industry hasn't converged yet on one standard way to describe what's happening inside an agent's context. Until that settles, teams instrumenting context today are, to some degree, betting on conventions that are still being written. Gartner projects more than 80%.