Context Window Management in Long-Running Enterprise Agents

How Context Actually Accumulates Across Dozens of LLM Calls

The basic loop is simple enough that it's easy to underestimate. Each LLM call ingests the original request plus all prior tool results, then passes its output forward. Context is additive. Nothing is automatically pruned. By the time an agent is twenty-five calls deep, the early context that defined the task's constraints and ground truth has either scrolled out of the active window or is competing with an enormous volume of intermediate results for the model's attention.

A coding agent makes this concrete. The agent writes a function at step five. By step twenty-five, that function's signature is buried under subsequent reasoning, new file reads, and tool responses. New code starts referencing a contract that no longer matches what was actually implemented. The agent doesn't know this, because it can't see step five anymore. The output looks right. It isn't.

Enterprise workflows make this worse in a specific, predictable way. Tool responses from systems like Microsoft Dynamics 365 Finance and Operations return hundreds to thousands of tokens of form metadata, navigation breadcrumbs, and schema state, most of it irrelevant to the next decision. It enters the context window anyway. It doesn't leave. Enterprise queries can consume 50,000 to 100,000 tokens before the model starts reasoning, pulling simultaneously from schema definitions, data lineage graphs, governance policies, and conversation history. A typical large engineering monorepo spans millions of tokens. A one-million-token window still can't hold a real organization's full context, so the question of what fills that window, and how fresh it is, matters more than the window's absolute size.

What Bigger Context Windows Do and Don't Solve

The expansion has been genuinely fast. GPT-3 launched with 2,000 tokens in 2020. Production models now routinely support 128,000 to one million tokens. Some real problems are solved by this: fewer hard cutoffs mid-task, room for richer system prompts, the ability to hold longer conversation history without immediate truncation.

But larger windows don't provide uniform attention across that space. Research on frontier models has found degradation at every increment of context growth, with the "lost-in-the-middle" effect causing accuracy drops exceeding 30% for information buried mid-context. The architectural reason isn't complicated once you see it. RoPE long-term decay reduces dot-product similarity between distant token pairs, and softmax normalization amplifies this by concentrating attention on the highest-scoring tokens. Primacy and recency dominate. The middle of a long context is functionally where information goes to be ignored, and this holds across model families with enough consistency that it should be treated as a structural constraint rather than a bug to be patched.

Cost compounds this in ways that only become visible at scale. Input tokens are priced per token, so doubling context size doubles input cost. Output tokens run at three to eight times input rates. Agentic workflows consume five to thirty times more tokens per task than a standard chatbot query. The economics of a poorly managed context window aren't abstract, and teams running large orchestrated deployments feel them in the infrastructure bill before they feel them in the evals.

The Four Degradation Patterns That Show Up in Production

Four patterns show up reliably in production systems. They compound each other, and any context management strategy worth deploying has to account for all of them, because fixing one while ignoring the others shifts the failure mode rather than eliminates it.

Context rot is the measurable decline in reasoning quality as the window fills, even when the token limit hasn't been reached. The model becomes less coherent not because it has run out of space but because the signal-to-noise ratio of the context has degraded. You can often see this in the outputs before you can explain it structurally, which makes it frustrating to diagnose.

Lost-in-the-middle is the most documented of the four. Research has demonstrated a U-shaped accuracy curve across multiple model families: information at the start and end of context is reliably used; information in the middle is systematically neglected. In a long-running agent, critical constraints and prior decisions often land in the middle of the window. They are not lost from the window. They are lost from the model's effective reasoning, which is a categorically different kind of loss and harder to catch because the model continues to respond fluently.

Context drift is subtler and often the most consequential in practice. Accumulated tool results and intermediate reasoning gradually displace the original task framing. The agent continues producing output. It is simply no longer solving the right problem. By the time this becomes visible, the agent has often taken dozens of consequential actions based on a misframed objective, and unwinding that is not straightforward.

Tool-response bloat is the upstream accelerant for everything else. Verbose system responses fill the window with irrelevant metadata faster than reasoning fills it with useful signal. This is particularly acute in expense management, CRM, and ITSM workflows, where each interaction returns form state and navigation breadcrumbs that have nothing to do with the decision at hand.

These four don't operate independently. Rot and drift worsen the lost-in-the-middle effect. Bloat accelerates all three.

Windowing and Observation Masking as the First Line of Defense

The simplest intervention is a rolling window: retain only the most recent N turns or tokens, discard older observations. No external infrastructure required. It directly addresses recency bias by ensuring the model works with the most recent state rather than a mixture of current and stale information, and for a lot of workflows, it's enough.

Observation masking adds something meaningfully different. Rather than deleting older observations entirely, masking replaces them with compact placeholders while keeping the agent's reasoning and actions intact. The model retains the logic of what it did without carrying the raw data bulk forward.

JetBrains' 2025 study compared masking against LLM summarization across 250-turn agent trajectories, and the finding was counterintuitive enough that it shifted how I think about these tradeoffs. Both approaches cut costs by over 50% compared to unmanaged contexts, but masking with Qwen3-Coder 480B achieved a 2.6% higher solve rate while being 52% cheaper than summarization. The reason matters: LLM summarization inadvertently extended agent trajectories by 13 to 15% by obscuring natural stopping signals. The summary was so good at compressing the work that it erased the evidence of completion. The agent kept working because it couldn't tell it had finished. That's a specific failure mode that summarization introduces, not one it inherits from the underlying task.

For tool-heavy, single-session workflows, a lightweight recency window plus a compact running summary is sufficient. The honest limit: windowing and masking discard history rather than preserve it. They fail when a long task genuinely requires retrieving something from early in the session, and no amount of tuning the window size fixes that.

Compression Strategies That Preserve Signal While Cutting Volume

Compression approaches fall into two broad families, and the choice between them depends more on what kind of information you're trying to preserve than on which sounds more sophisticated.

Natural-language compression, including abstractive and extractive summarization, is used in production in tools like OpenHands, Cursor, and Warp. It works well when the historical state is genuinely complex and a narrative summary can capture architectural decisions, constraints, and prior conclusions that would otherwise be lost. It works poorly for trajectory management, as the JetBrains finding makes specific. It also introduces a compressor model that can hallucinate or silently drop details, and those errors become the agent's blind spots, invisible by design.

Memory pointer approaches are structurally different in a way that matters. Instead of compressing the data, the agent stores it externally and passes a reference. The data is available when needed. It simply isn't present in the context window unless the agent explicitly retrieves it. Research on materials science validation workflows found this approach consumed approximately seven times fewer tokens than the traditional workflow. That's not a marginal efficiency gain; it changes what's economically viable to deploy.

The underlying principle is separating what the model needs to reason about now from what it merely needs to retrieve. Only the former belongs in the context window. Teams that implement this systematically end up with much cleaner contexts than teams that treat it as optional hygiene.

External Memory and Retrieval for State That Outlasts a Session

Windowing and compression work within a session. External memory handles state that must survive across sessions, agent restarts, or handoffs between specialized sub-agents. These are categorically different problems, and conflating them leads to systems that are undersized for the actual workflow they're running.

Retrieval-augmented generation is the established baseline, and most enterprise teams are already using some form of it. The limitation that matters for agentic workflows: conventional RAG assumes explicit queries and well-structured knowledge. Neither condition holds reliably when an agent mid-task is trying to retrieve something it doesn't have a precise name for. The agent knows approximately what it needs — different from knowing what to ask for — and standard RAG retrieval isn't designed for that approximation.

Newer dual-system architectures combine global memory formation with retrieval enhancement specifically for this case, where the query isn't cleanly stated and the agent needs to explore rather than look up. This matters more in practice than the architectural description suggests. An agent that can only retrieve what it can precisely articulate is a more limited agent than one that can probe memory in a more generative mode.

Production-grade dynamic memory systems extract, consolidate, and retrieve salient information rather than storing raw transcripts. The latency and cost differences versus full-context ingestion are substantial enough that they shape what's actually deployable at scale, not just what performs well on benchmarks.

Graph memory has moved from largely experimental to production-adjacent, though not because every agent needs a graph database. The shift is that memory systems are moving beyond pure vector similarity toward relational structure that captures how facts connect. For enterprise workflows, that relational layer carries real weight: knowing that a supply chain constraint is tied to a specific vendor's historical performance is different from retrieving both facts independently and hoping the model assembles the connection correctly.

One governance requirement that cannot be deferred: the retrieval layer must respect existing access controls. Answers should only pull from what the requesting user already has permission to see. Teams that treat this as a later-stage concern tend to discover the hard way that it isn't.

Planner-Executor Decomposition as a Structural Solution to Accumulation

Decomposition changes the math more fundamentally than any windowing or compression strategy, because it addresses accumulation architecturally rather than managing it after the fact. A high-level planner agent breaks a complex goal into bounded, verifiable subtasks and delegates each to a specialized executor. Executors return compact summaries, not full transcripts. The planner receives conclusions, not logs.

This reframes the accumulation problem entirely. Instead of one agent accumulating context across fifty tool calls, you have multiple agents each accumulating across five to ten calls. The total token budget across the system is similar, but no single window fills to saturation, and the planner's reasoning window stays clean because it never sees the raw tool responses, only the distilled results.

Enterprise teams working at scale tend to reserve 30 to 40% of the context window for accumulated work. On a 200,000-token model, a rough allocation is 80,000 tokens for system prompt and project context, 80,000 for tool results, and 40,000 for reasoning and output. Decomposition makes that allocation feasible by ensuring no single agent is expected to hold everything simultaneously. Without decomposition, these allocations become aspirational rather than operational.

The governance dimension of decomposed architectures is easy to overlook until it surfaces as a real problem. Decomposed systems require versioning and registry. A sub-agent or skill that works for one team shouldn't need to be rebuilt by another. Shared, governed registries prevent duplicated planner logic across departments and create a foundation for auditing which decisions were made and by which agent, which matters more in regulated industries than the architecture documents tend to acknowledge.

Tool-Response Discipline: Managing What Enters the Context Before It Becomes a Problem

Prevention is more efficient than remediation, and tool-response discipline is where prevention happens. Field selection, response schema design, and preprocessing filters can eliminate irrelevant metadata before it reaches the model. Navigation breadcrumbs, nested form state, and system headers from ERP responses are the primary targets.

In D365 F&O expense workflows, each tool response can contain thousands of tokens of form metadata for a task that needs only a handful of fields. A preprocessing or Model Context Protocol proxy layer strips this at the boundary. The model never sees the noise. It never has to work around it. The difference in downstream context cleanliness is significant enough that teams who implement this first often find their other context management problems shrink in scope.

MCP, introduced by Anthropic in late 2024 and now an open standard, provides session-based access control and lifecycle management that lets teams scope what data enters agent context per call. It is not a governance layer, and mistaking it for one is a common source of architectural regret. Teams still need to define which agents can call which tools, what data fields are returned, and how responses are filtered. MCP creates the plumbing. The decisions about what flows through it remain yours.

Every token entering the context window should either be used in the current reasoning step or retrievable from external memory. If neither condition applies, exclude it. Applied consistently, this eliminates the largest single source of bloat before it compounds downstream. Most production systems treat it as optional guidance rather than an enforced constraint, and the context bloat shows up accordingly.

Choosing the Right Combination for a Given Workflow Type

Session length, tool verbosity, task structure, and whether state needs to survive agent restarts: these four variables determine what context management stack is appropriate. Model capability and window size are secondary. Getting the stack wrong for the workflow type is how teams end up over-engineering short workflows or under-engineering long ones.

For short, tool-heavy, single-session workflows, such as expense itemization or ITSM ticket resolution, a recency window plus tool-response filtering is sufficient. No external memory store needed. The session is bounded, the task is relatively linear, and the primary failure mode is bloat from verbose tool responses, which filtering addresses at the source. Adding retrieval infrastructure to these workflows is over-engineering, and it introduces latency and complexity that don't pay off.

For long, branching, multi-session workflows like supply chain orchestration or multi-week engineering tasks, the appropriate stack is planner-executor decomposition plus external persistent memory with retrieval. Whether that retrieval layer uses dynamic extraction or graph memory depends on how relational the state is. Tasks where the connections between facts matter as much as the facts themselves benefit from relational structure. Tasks where facts are largely independent don't need it, and the added infrastructure cost won't justify itself.

The cost signal is a useful calibration instrument that doesn't require building a full evaluation framework before it's actionable. If inference costs are scaling faster than task complexity, the context management layer is undersized. The jump from a few cents per interaction several years ago to over a dollar for orchestrated agentic workflows today makes this signal visible in ways it wasn't when agentic deployments were small and mostly experimental.

What the trajectory and degradation research shows, taken together: default to observation masking over summarization for trajectory management, and treat summarization as a fallback for genuinely complex historical state rather than the first tool reached for. The JetBrains finding is specific enough that it should change the default for most teams, not serve as one data point among many.

One problem that doesn't have a clean answer yet: how do you verify that compression and retrieval haven't silently dropped a constraint or decision set early in the workflow? Evaluation and checkpointing for long-running agentic systems remain genuinely unsolved at the system level. Component-level solutions exist. An integrated approach to validation across a full multi-session workflow — one I'd actually trust in a regulated production environment — doesn't exist. Building agents in 2026 means holding that gap in view and designing around it deliberately, because the teams that assume it's been closed tend to find out otherwise at the worst possible moment.