Shouldn't I worry about output tokens? They cost 5x more.

Per token, yes. Per run, no. Loops read far more than they write, often 10 to 1, so input volume dominates the bill even at a fifth of the price. Chase input first: memory shape, anchors, caching. Trim output later, and carefully, because verbose reasoning in the planning step often pays for itself in fewer iterations.

Is accumulating context ever the right choice?

For short loops, yes. Under roughly ten iterations, the quadratic term hasn't woken up, and a continuous conversation can carry nuance that a state file flattens. The rule of thumb: if your budget allows more than ten iterations, build anchored from day one, because retrofitting memory shape after the invoice arrives is the expensive way to learn this lesson.

These prices are illustrative. Does the conclusion survive real pricing?

Yes, because the conclusion is about shape, not rates. Whatever your provider charges, the accumulating build multiplies input volume by the square of iterations while the anchored build keeps it linear, and cached input is dramatically cheaper than fresh input on every major provider. Change the prices and the curves move; their shapes, and the gap between them, do not.

How do I actually see my cache hit rate?

Most providers report cached versus fresh input tokens per request in the API response and billing exports; inference engines expose prefix-cache metrics directly. Log it per run at your gateway, alongside cost per run, and alert when a loop's hit rate drops: it almost always means someone edited an anchor file mid-run or reordered the context.

Why does an accumulating loop's cost grow quadratically?

Each iteration re-reads everything all previous iterations produced

Output gets longer every iteration

Each iteration re-reads everything all previous iterations produced

The model gets slower as context grows

Your 100-iteration loop is too expensive. Which change cuts the bill most?

Reset context each iteration and keep state on disk

Switch to a model that's 2x cheaper

Reduce output length per iteration

Reset context each iteration and keep state on disk

Run the loop during off-peak hours

Which metric decides whether a loop deserves to exist?

Cost per completed unit of work

← All posts

The Cost of a Loop: Token Economics of AI Agents That Run While You Sleep

Every loop engineering post warns about token costs in one nervous sentence, then changes the subject. This one does the math. The answer surprised us: the biggest cost lever isn't the model you pick. It's the shape of the loop's memory.

7/4/2026 · 5 min · jusCode · Read as Markdown

loop engineeringtoken economicsai agents

TL;DR

Loop cost is decided by the shape of the loop's memory, not the model you pick. An accumulating loop pays quadratically for linear work; an anchored, cached loop pays linearly and up to eleven times less.

Loop cost has three dials: iterations, tokens per iteration, and price per token. Each dial belongs to a different organ of the loop.
The Quadratic Trap: a loop that remembers by accumulating context pays quadratically for linear work. Same task, same model, up to 10x the bill.
Loops invert chat economics. Chat bills are output-heavy; loop bills are input-heavy, often 10 to 1, which makes prefix caching your largest discount.
Stop pricing tokens. Price outcomes: cost per run, cache hit rate, and cost per merged PR are the three numbers that belong on your dashboard.

The bill arrives

Your first loop invoice has a shape, and the shape is a parabola

The loop engineering discourse has a ritual. Every post celebrates agents that work while you sleep, then adds a single anxious line: watch your token costs. And then nothing. No model, no math, no numbers. Which is remarkable, because the numbers contain the single most useful fact about running loops in production, and it has nothing to do with which model you choose.

We covered the five organs of a loop and what loops do to the stack beneath them. Today we follow the money. To do it, we'll price one realistic loop three different ways and watch the same work produce three very different invoices.

The setup

One loop, three ways to pay for it

Take a real overnight workload: a loop that fixes a failing CI check. It runs 50 iterations before the gate goes green. Its anchor context (system prompt, spec, conventions doc) is 10,000 tokens. Each iteration generates about 2,000 output tokens and produces 4,000 tokens of new material: diffs, test logs, tool results. Price it at an illustrative frontier-class rate of $3 per million input tokens and $15 per million output tokens, with cached input at a typical tenth of the input price.

The output side of this bill is fixed: 50 iterations times 2,000 tokens is 100,000 output tokens, which costs $1.50 no matter what you do. Everything interesting happens on the input side, and the input side is decided by one architectural choice: what does the loop remember, and how?

Way 1: the accumulating loop

The naive build keeps one growing conversation. Iteration 1 sees the 10k anchor. Iteration 2 sees the anchor plus everything iteration 1 produced. By iteration 50, the loop is re-reading 49 iterations of history just to take its next step. Sum it up and this loop reads 7.85 million input tokens: about $25 per run.

Way 2: the anchored loop

The disciplined build resets context every iteration and reloads only the anchor plus the latest delta and a compact state file. Every iteration reads roughly 14k tokens, flat. Fifty iterations read 700,000 input tokens: about $3.60 per run. Same model. Same 50 iterations. Same green checkmark at the end. Seven times cheaper.

Way 3: anchored plus cached

Now switch on prefix caching. The 10k anchor is identical on every iteration, so after the first pass it's billed at the cache rate. Fresh input drops to the 4k delta, and the run lands around $2.25: eleven times cheaper than the accumulating build, with zero loss of capability.

Fifty iterations of identical work, priced three ways. The gap is not the model. It is the memory.

The unique insight

The Quadratic Trap

Here's why the accumulating loop's curve bends upward. When context grows by a fixed amount every iteration, iteration N re-reads everything from iterations 1 through N minus 1. Total input across the run is the sum of a growing series, and that sum grows with the square of the iteration count. Double the iterations, quadruple the input bill. This is the Quadratic Trap: a loop that remembers by accumulating pays quadratically for linear work. It's also why the trap stays hidden in demos. At 5 iterations, all three builds cost pennies and look identical. The parabola only reveals itself at 30, 50, 100 iterations, exactly when you've stopped watching. And notice what does not fix it: a model that's three times cheaper still leaves you on a quadratic curve, just a slightly flatter one. Price cuts move the curve. Memory shape changes its equation.

ACCUMULATING MEMORY

Pays O(N²)

context grows every turn. Every iteration re-buys the entire past at full price.

ANCHORED MEMORY

Pays O(N)

progress lives on disk. Each iteration buys only the anchor (cached) and what's new.

The receipt

What one good run actually costs

The anchored-plus-cached build, itemized. Print this shape into your observability dashboard.

LOOP RUN RECEIPT fix-failing-ci · 50 iterations · overnight

anchor prefix (cached) 500k tok$0.15

fresh input (deltas) 200k tok$0.60

output 100k tok$1.50

TOTAL PER RUN$2.25

same run, accumulating build$25.05

outcome1 merged PR

cost per merged PR: $2.25 · your move, finance

The inversion

Loops are input machines

One more thing the receipt reveals. In chat products, the expensive line is output: users ask short questions and models write long answers, and output tokens cost five times more per token. Loops invert this completely. Our run read 700,000 tokens to write 100,000. Even at a fifth of the per-token price, input is where the volume lives, often 10 to 1. The practical consequence: every optimization that targets input is worth five that target output. That's why anchor files, context ordering, and prefix caching [1][2] dominate loop economics, and why the serving research community has spent years making repeated input nearly free [3]. Loops are the workload that research was waiting for.

The loop P&L: three numbers, one dashboard

Cost per run: the unit finance actually experiences. Cache hit rate: below 80 percent on a mature loop means your anchors are unstable or your context ordering is wrong. Cost per merged PR (or per completed unit of work): the only number that decides whether the loop deserves to exist. A $4 run that lands a PR beats a $0.40 run that lands nothing, every single time. Stop pricing tokens. Price outcomes.

References

The research behind the discount

Kwon et al., 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 23). The memory management that makes prefix reuse practical at scale. arXiv:2309.06180
Zheng et al., 2024. SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 24). RadixAttention: KV-cache reuse across the repeated calls of an LLM program, the exact shape of a loop. arXiv:2312.07104
Pope et al., 2022. Efficiently Scaling Transformer Inference. The foundational analysis of where inference cost actually comes from. arXiv:2211.05102

Part of our loop engineering series: The Anatomy of an Agent Loop · Loops Are Layer 04 · The 11 Layers of the AI Inference Stack. Next: what loop engineering means for the people signing these invoices.

Test yourself

1. Why does an accumulating loop's cost grow quadratically?
2. Your 100-iteration loop is too expensive. Which change cuts the bill most?
3. Which metric decides whether a loop deserves to exist?

Shouldn't I worry about output tokens? They cost 5x more.: Per token, yes. Per run, no. Loops read far more than they write, often 10 to 1, so input volume dominates the bill even at a fifth of the price. Chase input first: memory shape, anchors, caching. Trim output later, and carefully, because verbose reasoning in the planning step often pays for itself in fewer iterations.
Is accumulating context ever the right choice?: For short loops, yes. Under roughly ten iterations, the quadratic term hasn't woken up, and a continuous conversation can carry nuance that a state file flattens. The rule of thumb: if your budget allows more than ten iterations, build anchored from day one, because retrofitting memory shape after the invoice arrives is the expensive way to learn this lesson.
These prices are illustrative. Does the conclusion survive real pricing?: Yes, because the conclusion is about shape, not rates. Whatever your provider charges, the accumulating build multiplies input volume by the square of iterations while the anchored build keeps it linear, and cached input is dramatically cheaper than fresh input on every major provider. Change the prices and the curves move; their shapes, and the gap between them, do not.
How do I actually see my cache hit rate?: Most providers report cached versus fresh input tokens per request in the API response and billing exports; inference engines expose prefix-cache metrics directly. Log it per run at your gateway, alongside cost per run, and alert when a loop's hit rate drops: it almost always means someone edited an anchor file mid-run or reordered the context.