External State Stores for Stateless Agent Runtimes
Stateless runtimes need external stores to persist agent memory across calls.
A single inference call is a closed loop. Tokens go in, attention runs across the full input sequence, an output comes out, and the call ends. Every intermediate representation the model built along the way gets thrown away the moment that response is sent. Nothing about that call gets written to disk, and nothing from it carries forward into the next one. That is how the model was built to run.
Two things get confused constantly here, and they need to stay separate. Model weights are the frozen patterns the model learned during training. They do not change at inference time, no matter how long a conversation runs or how many calls happen in sequence. Session state is a different thing entirely: the conversation history, the documents retrieved for context, the user's stated preferences. None of that lives inside the model. The model holds weights. It does not hold a memory of the user.
That separation is what makes horizontal scaling possible. A stateless inference instance is interchangeable with every other instance running the same weights, so any pod can answer any request that comes in. Load balancers do not need sticky routing to keep a user tied to one machine. A crashed instance gets replaced without losing data, because there was never any data sitting on that instance to lose. These properties are the foundation that lets AI services run at the volume production systems actually demand.
The clearest confirmation of this direction comes from the Model Context Protocol itself. The MCP specification release candidate dated 2026-07-28 removes transport-level session management entirely, leaving agents with a stateless protocol core that runs on ordinary HTTP load-balanced infrastructure. The older, stateful version of that protocol had real costs in production. Session state pinned to a specific pod meant standard load balancers threw session-not-found errors whenever a request landed somewhere else. Teams had to configure sticky session affinity rules just to keep sessions alive, which meant traffic could no longer distribute evenly across a fleet. A pod restart wiped out any in-flight session instantly. Running a remote MCP server meant choosing between sticky affinity rules, a shared Redis session store, or gateway-level packet inspection, and all three options added latency and operational overhead. The GitHub MCP Server has already moved to the new spec and removed its Redis session storage entirely, cutting out a database read and write on every single call. The protocol layer is moving toward statelessness because statelessness is what scales. That is the premise the rest of this piece builds on.
What statelessness breaks when agents run real work
Statelessness holds up fine as long as a task fits inside one call. The trouble starts the moment agent work spans more than one request, because a stateless runtime forgets everything the instant the call ends, and for any task with a lasting identity, that forgetting is a structural failure, not a minor nuisance.
Chat tools that appear to "remember" a conversation are not relying on any native memory inside the model. They are re-injecting the prior turns of the conversation as tokens at the start of every new request. The model processes that full history as input each time. It never retrieves anything from a persistent state of its own, because it has none. That re-injection approach works, but it does not scale cleanly. Every additional turn adds more tokens that need to be re-sent, and the growth compounds: resending the full history on every turn produces token counts that climb quadratically as a conversation lengthens, and the latency and cost climb right along with it. At enterprise scale, that compounding is not a rounding error. It becomes the dominant cost in the system.
None of this means stateless agents are the wrong tool across the board. Plenty of real work fits comfortably inside a single call. Summarization, extraction, and classification tasks are atomic: the entire relevant context fits in one request, and nothing in the task has an identity that needs to persist afterward. High-throughput batch pipelines, the kind running content moderation or generating embeddings at volume, benefit directly from statelessness, since horizontally scalable workers carry zero session overhead between jobs. Certain compliance contexts actually require statelessness as a guarantee: a runtime that writes no conversation data to disk satisfies data-residency requirements that a stateful system cannot promise as cleanly.
Real work rarely stays this contained, though. It spans a morning, touches a customer record, and crosses three different systems before it is done. Something has to hold the thread connecting those steps, decide which tool to call next, and stop an agent from retrying a payment five times because it lost track of whether the first attempt succeeded. That something is the runtime, and without it, every failure point forces a restart from zero. The risk compounds further in persistent shared environments, where state accumulates silently across sessions. A malformed or malicious run earlier in a session's life can quietly influence behavior much later, since nothing cleared the slate in between. Ephemeral environments fix that specific risk by design, wiping the slate after every run. But that fix only works if whatever actually mattered from that run got written somewhere external before the environment disappeared. Otherwise the agent is just forgetful in a different, more dangerous way, because now the thing that mattered is gone along with the thing that was risky.
The three-tier state model that production agent systems converge on
By 2026, production agent architectures had largely converged on the same answer to this problem, and it is not to make the model stateful. It is to keep the model stateless, keep the runtime stateful, and give the state itself a home outside both of them, organized into three tiers that get projected back into each call.
The first tier is working memory: the active context window for a single call, holding the current message buffer, the tool traces from that call, and the system prompt. This tier does not need a persistent store of its own. The runtime reconstructs it fresh for every call from the two tiers beneath it. The real question for this tier is not where it lives but how quickly it can be assembled, since every millisecond spent retrieving the material that fills it is a millisecond added to the time before the model can even start reasoning.
The second tier is episodic, or session, memory: the time-ordered log of what happened earlier in this particular task, including conversation history, the outcomes of tool calls, and intermediate reasoning steps. This tier lives in external storage and gets loaded back in at the start of each new call within the same session.
The third tier is long-term, or semantic, memory: knowledge, preferences, and resolved experience accumulated across every session the agent has ever run, not just the current one. Unlike the context window, which has a hard ceiling, external memory modules at this tier carry a capacity that is effectively unbounded, so an agent's cross-session memory, its experiential data, and its accumulated domain knowledge can all persist there indefinitely.
Splitting state this way is not an aesthetic preference. Each tier carries a different latency tolerance, a different consistency requirement, and a different cost profile. Collapsing all three into a single store, or worse, trying to cram all of it into the context window itself, trades correctness for simplicity, and that trade fails the moment load increases.
Each tier's substrate follows directly from the access pattern it needs, not from which database happens to be popular. Session memory needs fast reads for short-lived coordination, which is what a key-value store like Redis is built for: low latency, a simple data model, well suited to turn-by-turn conversation state where the window in play is bounded. But session records that need to be durable and queryable across sessions, not just fast, belong in a relational database. The MemoryOps pattern stores structured session data such as incident IDs, services, severity, status, logs, and metrics directly in PostgreSQL, which gives transactional guarantees that a pure key-value layer does not. The pattern that holds up in practice is to keep durable workflow truth in PostgreSQL and reach for a fast key-value layer only where latency genuinely demands it. Engineering teams running AI applications have been collapsing multi-database stacks back into Postgres for exactly this reason: it now handles workloads that previously required four separate tools, including pgvector-backed agent memory with the same transactional guarantees as the rest of the data.
That consolidation carries over into the third tier as well. Vector databases, whether that is pgvector or a dedicated vector store, serve semantic retrieval, letting the agent query by meaning instead of by exact key and pull back whatever past experience is most relevant to the task at hand. The cost logic here is directional but significant: cramming all relevant history into the context window scales its cost with the size of the accumulated corpus, while retrieval cost stays flat no matter how large that corpus grows. The gap between those two curves widens the longer an agent operates. Structured stores and knowledge graphs round out this tier, holding consolidated facts, preferences, and profiles that call for deterministic lookup.
Splitting state across three or four systems does raise a real concern: more moving parts means more latency and more operational complexity to manage. One documented failure mode shows how that complexity bites: an agent reasons correctly over context that is simply stale, because a store's index was never refreshed after the underlying source documents changed. The answer to that is stricter write discipline and firmer guarantees around index freshness, so that every tier reflects what is actually true at the moment it gets queried.
The runtime's projection of stored state back into each call
The runtime does not hold state itself. Its job is to retrieve the right slice of external state at the right moment and assemble it into a context window that makes the model's next call coherent with everything that came before it. That assembly happens in five steps, every single time a call is made.
The runtime first resolves the session: it receives the incoming request, identifies which session or agent identity it belongs to, and looks up the relevant episodic state from the session store. From there it moves into semantic retrieval, querying the long-term store for experience and knowledge relevant to the current task, using the user's input or the current task state as the query itself. With both of those pulled, the runtime assembles context: the system prompt, the retrieved long-term material, the recent session history, and the current input all get arranged into the context window in a specific order, fit within the token budget, with the most relevant material given priority. Only then does the model call happen. The assembled context goes to the stateless model, and the model has no idea any of it was pulled from external storage moments earlier. Once the model responds, the runtime writes back: new observations, tool-call outcomes, and reasoning steps get written to the episodic store, and when a trajectory finishes, consolidation logic writes the durable facts worth keeping into the long-term store.
The Microsoft Agent Framework's FileMemoryProvider shows this pattern running in a live system. Foundry Hosted Agents package the agent as a container on managed infrastructure, with built-in automatic scaling, managed session state, and observability. Sessions resume with the filesystem fully intact: files, disk state, and session identity all persist across scale-to-zero, so the agent picks back up where it left off.
One detail in that write-back step carries real weight. The runtime has to know which steps are safe to repeat and which are not. Retrying a read costs nothing. Retrying a write can duplicate a charge or send a customer the same email twice. State write-back is not idempotent by default, so the runtime has to track which writes already landed before it decides whether to retry anything.
That tracking is also what lets the system know when a task is actually finished. Done, in a stateful session, means a business outcome has been reached, not that a token stream stopped generating text. The runtime needs to distinguish a completed job from one that is waiting on a person, and both of those from one that has failed in a way a human now has to see. None of those three states are visible from a single call in isolation. They only become visible because the external store carries the thread across the whole session. Ephemeral execution environments depend on this same discipline: an environment gets created fresh for each task and destroyed afterward, so a compromised session cannot persist access or bleed into the next run. That protection only holds if whatever mattered from the session was written to the external store before the environment was torn down.
The context-window-expansion objection
If context windows keep getting larger, a reasonable question is why not just load the entire relevant history into the prompt and skip external storage altogether, and this question deserves a direct answer.
A bigger context window changes the scale of the problem. It does not change its shape. In-context memory cost grows with the size of the corpus being loaded, while retrieval cost stays flat no matter how large that corpus gets. A larger window pushes the point where that gap becomes painful further out, but the gap itself does not close, and an agent that accumulates experience across thousands of sessions will eventually outrun any fixed window no matter how generous it is.
Scaling properties matter just as much as raw capacity. A stateless runtime scales horizontally because any pod can serve any request, and that only holds true as long as the state those requests depend on lives outside the call, in a store every pod can reach. Load the relevant history directly into every request through an enormous context window instead, and the compute cost of attention over that history scales with the window's size on every single call, even for turns where most of that history is irrelevant. Retrieval exists precisely to avoid paying that cost repeatedly for information that only a fraction of each call actually needs.
There is also the matter of what a bigger window does not provide: durability, write discipline, and structured query access. A context window is not queryable the way a relational table or a vector index is. It cannot be partially updated. It cannot enforce a transactional guarantee the way PostgreSQL can. It cannot be inspected by a separate process checking for staleness, the way an index freshness check can run against a vector store. A context window is a staging area the runtime assembles right before a call. It was never built to be the record of truth, and expanding its size does not change that role.
External state stores earn their place because they solve a problem that context length alone cannot touch: where the durable record of a task lives, how it stays current, and how cheaply it can be found again when the next call needs it.
