Loops Are Layer 04: Where Loop Engineering Fits in the AI Inference Stack
The loop engineering debate treats loops as a workflow trick. Your infrastructure sees something else: the first customer that generates demand at machine speed. Here is what every layer must do differently, and the cache trick that flips the economics.
7/3/2026 · 6 min · jusCode · Read as Markdown
TL;DR
A loop is an amplifier: one human intention becomes fifty to a hundred model calls. That makes loops the first machine customer of your inference stack, and, counterintuitively, the most cacheable traffic you have.
- A loop is an amplifier: one human intention becomes 50 to 100 model calls before the gate says done.
- Loops are machine customers. They never sleep, never hesitate, and never get bored, so your traffic stops following the sun.
- The counterintuitive part: loop traffic is the most cacheable traffic you have. Iteration N shares almost its entire prefix with iteration N minus 1.
- Five layers change when loops arrive: gateway budgets per run, router routes per step, context anchors on files, cache becomes the profit center, serving learns backpressure.
The shape of the problem
The day your first loop ships, your traffic graph changes shape
For two years your inference traffic has followed the sun. It climbs when engineers sit down, dips at lunch, and dies at night, because every request began with a human typing. Then someone on your team reads about loop engineering, schedules an agent to fix failing CI checks overnight, and your dashboard grows a second heartbeat. Peak traffic at 3am. Nobody typed anything.
In our dissection of the agent loop we covered the five organs that make a loop trustworthy. This post is about the other side of the relationship: what the loop does to the infrastructure underneath it. In the 11 layers of the inference stack, loops live at Layer 04, the orchestrator. And when Layer 04 starts running hot, every layer around it discovers it was designed for a customer that no longer exists: a human, typing at human speed.
The multiplier
Do the math before your invoice does
Start with one intention: "fix the failing test in this repo." A well-built loop turns that into roughly 15 iterations before the gate goes green. Each iteration is not one model call. It is a plan step, two or three tool calls, and a verification pass. Call it five. One intention, seventy-five calls. Now schedule that loop across twenty repos overnight, and your single sentence of intent became fifteen hundred model calls while you slept.
This is not a bug. It is the entire point of a loop: trading cheap machine iterations for expensive human attention. But it means the question "what does a request cost?" quietly became "what does a run cost?", and most infrastructure cannot answer that yet.
Layer by layer
What changes when loops arrive
Five layers of the stack feel loop pressure first. Here is the specific adjustment each one needs.
Budget per run, not per user
Per-user rate limits were designed to stop a human from hammering refresh. A loop is one user generating a thousand legitimate calls. The gateway needs a new unit of account: the run. Give every loop run an identity, a spend ceiling, and a kill switch. When something goes wrong at 3am, you want to revoke one run's key, not your team's.
what changes: quotas move from "calls per user per minute" to "spend per run, with a hard stop."
Route the step, not the task
Inside one loop, the iterations are not equal. The planning step that decides what to do next deserves your strongest model. The mechanical steps, applying a diff, summarizing test output, formatting a commit message, do not. A router that sees "coding task, frontier model" wastes most of its spend. A router that sees steps can send each one to the cheapest model that clears the bar, and the loop's own gate catches any quality slip for free.
what changes: routing granularity drops from the task to the individual iteration step.
Anchor files are a context strategy
The best loops reset context every iteration and reload the same anchor files: the spec, the state file, the conventions doc. That practice, born as a reliability trick, is secretly a context-layer decision with enormous consequences one layer down. Stable, identical context at the start of every iteration is exactly what the next layer needs to work its magic.
what changes: context assembly optimizes for stability across iterations, not just relevance within one.
The layer that flips the economics
Here is the insight this post exists for, so we gave it its own section below. Short version: the traffic that looks scariest on your invoice is the traffic your cache loves most.
what changes: the cache goes from a nice-to-have to the difference between loops you can afford and loops you cannot.
Backpressure meets a customer that never blinks
Human traffic backs off naturally: people see a slow response and pause. A loop does not. It fires its next iteration the instant the last one returns, and twenty loops firing together produce square-wave load your autoscaler has never seen. The serving layer needs explicit backpressure that loops respect: queue-depth signals, per-run concurrency caps, and priority so overnight loops never starve a human waiting on a keystroke.
what changes: machine traffic gets its own queue and its own priority class, always below humans.
The unique insight
Loop traffic is the most cacheable traffic you have
Think about what iteration 7 of a loop actually sends: the same system prompt as iteration 6, the same spec, the same anchor files, the same conventions doc, plus one small delta of new observations. The prefix overlap between consecutive iterations routinely exceeds 90 percent. Human conversations never look like this; every user phrases things differently. Loops repeat themselves by design. Modern inference engines exploit exactly this: prefix and KV-cache reuse means the shared portion of the prompt is computed once and replayed nearly free on every following iteration [1][2]. The conclusion inverts the fear that dominates the loop debate. Uncached, a 75-call loop pays full price 75 times and deserves its scary reputation. With prefix caching and stable anchor files, most of every call is a cache hit, and cost per iteration collapses. The loop did not get cheaper by thinking less. It got cheaper by repeating itself to a stack that rewards repetition.
Seeing it
Three iterations, one prefix
What the cache sees when a well-anchored loop runs.
Four checks, one per layer: a per-run budget with a kill switch at the gateway. Step-level routing so mechanical iterations run cheap. Anchor files so context stays stable across iterations. Prefix caching on, verified with a cache-hit metric on your dashboard, because a loop without cache hits is a loop you are overpaying for.
References
The research behind the cache claim
- Zheng et al., 2024. SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 24). Introduces RadixAttention, KV-cache reuse across the multiple calls of an LLM program, which is precisely the shape of a loop. arXiv:2312.07104
- Kwon et al., 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 23). The vLLM paper: paged KV-cache management that makes prefix sharing practical at scale. arXiv:2309.06180
- Yao et al., 2023. ReAct: Synergizing Reasoning and Acting in Language Models (ICLR 23). The loop body itself: reason, act, observe, repeat. arXiv:2210.03629
Part of our loop engineering series. Previously: The Anatomy of an Agent Loop and The 11 Layers of the AI Inference Stack. Next: the full token economics of loops that run while you sleep.
Test yourself
1. Why is loop traffic unusually cache-friendly?
2. What is the right unit for a gateway budget once loops exist?
3. Which loop step genuinely deserves your strongest model?
FAQ
- Doesn't the cache break every time the loop edits a file?
- Only the changed part recomputes. Prefix caching works front to back: the system prompt, spec, and conventions doc at the start of the context stay identical across iterations, so they stay cached. Put the volatile material (latest diff, test output) at the end of the prompt and you preserve the long stable prefix. Context ordering is now a cost decision.
- Should every iteration use the frontier model to be safe?
- No, and this is what makes loops special: the gate protects you. If a cheap model botches a mechanical step, the tests fail and the loop retries. You pay one extra cheap iteration instead of a frontier premium on every step. Reserve the strong model for planning, where a bad decision costs many iterations downstream.
- Won't overnight loops explode my GPU bill?
- Unbudgeted and uncached, yes, and you will find out from finance. With the four checks in the callout above (per-run budget, step routing, anchor files, prefix caching) the cost per unit of completed work usually lands below what interactive usage costs for the same outcome, because nothing about a loop requires a human's expensive attention while it runs.
- Is this only about coding agents?
- Coding is where loops matured first because the gates are free: compilers and test suites. But the infrastructure math is identical for any looped work, research digests, data pipeline repair, document processing. Wherever a loop runs, Layer 04 amplifies demand and the layers below feel it.