Bigger, but Slower: Optimal Context Windows for LLM Agents

1. Introduction

The rapid expansion of context windows has become an important dimension of progress in large language models. Contemporary models increasingly support context windows measured in hundreds of thousands, and sometimes millions, of tokens. Agent harnesses consequently tend to treat larger usable context as unambiguously desirable. A common strategy is to allow the conversation history to grow until it approaches a fixed proportion of the model’s maximum context window, such as 80%, and only then compact or summarize the history.

This approach implicitly equates context capacity with context efficiency.

For long-running local agents, these quantities can diverge substantially.

An agent does not merely store its context. It must process that context. As the working context grows, prefill becomes an increasingly important component of end-to-end latency. More importantly, prefill throughput itself may decline as context grows. Consequently, an agent operating at 100,000 tokens may spend considerably more time processing each additional unit of work than the same agent operating at 30,000 or 50,000 tokens.

Decode throughput alone does not capture this effect. A local model capable of generating 50 tokens per second can nevertheless feel progressively slower during a long agent session if increasingly large portions of execution time are spent processing context before generation begins.

The obvious response is to compact more frequently. Yet compaction is not free. A conventional model-based compaction procedure may itself require substantial inference. Even when compaction is performed cheaply using deterministic or CPU-based methods, the compacted working context must subsequently be processed again to reconstruct the model’s inference state. Excessively frequent compaction can therefore waste substantial computation repeatedly rebuilding a minimum viable working context.

This produces two opposing forces.

Increasing the working-context threshold reduces the frequency of compaction and context reconstruction:

At the same time, it forces the model to spend a larger fraction of its working life at longer, slower contexts:

We argue that these opposing forces create an interior optimum. For a given model, hardware configuration, inference backend, and compaction strategy, there exists a working-context threshold that minimizes expected task completion time.

We call this quantity the optimal working context.

The distinction is fundamental:

The maximum context (C_{\max}) describes what the system can accommodate. The optimal working context describes where the system should operate.

This paper develops a simple analytical framework for estimating this point and proposes a practical mechanism through which an agent harness can learn it passively from ordinary usage.

2. Context Length as a Runtime Cost

Consider a long-running agent that accumulates context as it performs work. Let (c) denote the current working-context length and let

denote prefill throughput, measured in tokens per second, at context length (c).

A useful first-order empirical model is

where (S_0) represents short-context prefill throughput and (K) controls the rate of degradation with context length.

The parameter (K) can be interpreted as a context-length degradation scale. At

the model predicts

Larger values of (K) therefore characterize systems that preserve prefill throughput more effectively as context grows.

This functional form is not essential to our argument. In practice, (S(c)) can be estimated nonparametrically from observed runtime data. We use the analytical form because it makes the underlying optimization transparent.

The time required to process an infinitesimal amount of context (dc) is

The time required to build a context of length (H) from an empty inference state is therefore

Under the degradation model above,

The quadratic term is important. Even though the context itself grows linearly, cumulative processing cost can grow superlinearly because later tokens are processed under progressively less favorable prefill conditions.

3. Stable Agent Operation With Compaction

We next consider an agent operating for a sufficiently long period that it reaches a stable compaction regime.

Let

denote the context retained after compaction. (R) represents the minimum viable working context required to preserve instructions, task state, recent interaction history, relevant tool state, and other information necessary for continued operation.

Let

denote the compaction threshold.

A stable operating cycle is therefore

During each cycle, the agent performs

tokens of new effective work before another compaction is required.

After compaction, however, the resulting context must be processed again. The system therefore repeatedly pays the cost of reconstructing approximately (R) tokens of working state.

This rebuild cost is central to the optimization problem. If it were zero, the time-minimizing strategy would simply be to compact continuously and maintain the smallest possible context. Once reconstruction is included, excessively small context thresholds become inefficient.

4. Time per Unit of Effective Work

Suppose first that the computational cost of producing the compacted representation itself is negligible. This approximates deterministic or CPU-based compaction whose runtime is small relative to GPU inference.

One complete cycle processes context from an empty inference state through (H), at cost

Yet this cycle produces only

tokens of new effective work.

The average prefill time per unit of new work is therefore

This is the central objective function.

As

the denominator approaches zero. The system repeatedly rebuilds a large retained context while accomplishing almost no new work. Average cost therefore becomes very large.

At the opposite extreme, increasing (H) allows rebuild cost to be amortized over more new work, but increasingly large portions of the cycle occur at context lengths where prefill throughput is degraded.

The resulting objective is U-shaped.

The optimal compaction threshold is

Differentiating gives

Several implications follow immediately.

First, the optimal threshold is not a fixed proportion of maximum context capacity.

Second, the optimal threshold depends on the minimum retained working context (R).

Third, it depends on the machine-specific degradation parameter (K).

Fourth, short-context throughput (S_0) scales total execution time but does not determine the optimum under this simplified model. Two systems with different absolute prefill speeds can therefore share the same optimal context threshold if their relative degradation functions are identical.

5. General Form

The analytical degradation function above is useful for exposition but unnecessary for implementation.

Let the empirically observed prefill function be

Let

denote the observed time required to reconstruct a post-compaction context of size (R), and let

denote the computational cost of compaction itself.

Then the average time cost per unit of effective work is

The optimal threshold is simply

This formulation requires no assumption about the mathematical shape of prefill degradation.

It also accommodates hardware effects, inference-engine behavior, quantized KV caches, model architecture, and other implementation-specific factors automatically through the observed function (\hat S(c)).

6. Learning the Optimal Context From Usage

Dedicated benchmarking is not necessary.

An agent harness already observes the information required to estimate the optimization function during normal operation. Each inference request can generate an observation

where (c_i) is the context length and (s_i) is observed prefill throughput.

Over time, these observations provide an empirical estimate of

Measurements should be associated with a specific runtime profile because prefill behavior is not solely a property of the GPU. A useful profile identifier includes

A change from Q3 weights with Q4 KV cache to Q4 weights with Q8 KV cache, for example, should generally create a different profile.

Observed measurements can be aggregated into context buckets, such as 5K or 10K token intervals. Robust statistics such as medians can reduce contamination from transient system load, unusual tool outputs, cache effects, or concurrent processes. Alternatively, exponentially weighted estimates can allow the profile to adapt gradually to software or runtime changes.

The harness can similarly observe actual post-compaction reconstruction latency and estimate (B(R)).

At shutdown, periodically during execution, or after a sufficient number of observations, these statistics can be persisted.

At the next startup, the harness evaluates

over feasible values of (H) and selects

The threshold remains fixed during the subsequent runtime session, avoiding unstable online policy changes while allowing the system to learn continuously across sessions.

7. From Capacity-Aware to Throughput-Aware Compaction

Many existing context-management policies can be characterized as capacity-aware:

where (\alpha) may, for example, equal 0.8.

Such a rule answers the question:

How much context can safely be consumed before the model reaches its capacity limit?

It does not answer:

At what context length does continued accumulation become slower than compacting and rebuilding the working state?

These are different optimization objectives.

We therefore distinguish three quantities:

and

A model might support 128K tokens while the observed throughput-optimal threshold on a particular machine is 82K. The remaining capacity is still valuable as headroom for exceptional requests, but routine operation need not consume it.

This distinction becomes particularly important for local inference, where hardware configurations vary dramatically and prefill degradation may differ substantially across GPUs, unified-memory systems, quantization strategies, and inference backends.

8. Evaluation

The hypothesis can be tested using fixed-workload long-horizon agent tasks.

For each hardware and model configuration, the experiment should hold constant the total effective workload and output requirement while varying the compaction threshold (H).

For example, an agent can be required to process a fixed cumulative workload while thresholds are independently set to

For each condition, we measure:

1. total wall-clock completion time;
2. cumulative prefill time;
3. cumulative decode time;
4. number of compaction events;
5. cumulative context reconstruction time;
6. effective prefill throughput as a function of context length;
7. task completion quality.

The primary prediction is a U-shaped relationship between the compaction threshold and total completion time.

A second test compares the empirically observed minimum with the threshold predicted from passively learned historical telemetry.

A third test compares throughput-aware compaction against fixed-percentage baselines such as 50%, 70%, and 80% of maximum context.

The key outcome is not whether a system can technically support a larger context. It is whether operating at that context improves total task productivity.

9. Implications for Local LLM Benchmarking

Local LLM performance is commonly summarized by decode throughput. This metric is useful but increasingly incomplete for agent workloads.

An interactive agent repeatedly performs prefill, tool execution, context accumulation, state reconstruction, and decode. As sessions grow longer, prefill can become a dominant source of latency even when decode remains fast.

A GPU producing 60 output tokens per second may therefore complete a long-running agent task more slowly than another GPU producing 45 tokens per second if the latter maintains substantially higher long-context prefill throughput.

Maximum context capacity creates a similar measurement problem. A system capable of holding 150K tokens is not necessarily more productive when operated at 150K tokens.

We therefore propose time-optimal working context as an additional systems metric for local and agentic inference.

More broadly, hardware evaluation for LLM agents should consider the complete workload:

The relevant performance question is ultimately not how rapidly the model emits tokens, but how rapidly the system completes useful work.

10. Discussion

The framework produces a counterintuitive conclusion: larger context windows can reduce agent productivity even when sufficient memory is available.

This does not imply that large context capacity is undesirable. Greater capacity provides flexibility, supports workloads that genuinely require large contiguous contexts, and creates safety margins against overflow. Rather, capacity and operating point should be treated separately.

The analogy is similar to other computing resources. A processor’s maximum clock frequency is not necessarily its most efficient sustained operating point, and the maximum amount of allocatable memory is not necessarily the amount an application should continuously consume.

Context should similarly be treated as a managed runtime resource.

This perspective also changes the role of compaction. Compaction is not merely an emergency mechanism invoked when the model approaches its context limit. It can be an ordinary scheduling operation used deliberately to keep inference within a favorable performance region.

The optimal policy depends on both computation and information. Aggressive compaction can improve runtime while degrading task quality by discarding information required later. Consequently, a complete optimization objective should eventually incorporate an information-loss penalty:

where (L_{\mathrm{information}}) represents the expected downstream cost of information lost through compaction.

The purely time-optimal threshold developed in this paper is therefore a necessary first step rather than the final context-management objective.

11. Conclusion

Longer context is not free.

For long-horizon LLM agents, allowing working context to grow reduces compaction frequency but can progressively degrade prefill throughput. Compacting too aggressively creates the opposite problem by repeatedly requiring the system to reconstruct a substantial minimum working context.

These opposing costs imply a time-optimal working-context threshold.

Rather than selecting compaction thresholds as a fixed percentage of maximum model context, agent harnesses can learn the relevant prefill degradation function and reconstruction cost from ordinary inference telemetry. The resulting threshold is specific to the actual model, hardware, quantization configuration, backend, and workload.

The central distinction is therefore simple:

For long-running LLM agents, bigger is not always faster.