The 11 Layers of the AI Inference Stack: From Prompt to Silicon
Say inference stack to three engineers and you get three different buildings. Follow one request from prompt to silicon and it resolves into 11 layers, with a line showing which ones are yours to own.
7/2/2026 · 2 min · jusCode · Read as Markdown
TL;DR
Inference stack is ambiguous. Follow one request from prompt to silicon and it passes through 11 distinct layers. Observability is a vertical across all of them, not a layer. You do not need to own every layer; you need to know where your line is. Ours sits just above the inference engine.
- Inference stack is ambiguous: infra teams mean everything below the serving engine, app teams mean everything above it.
- The fix: follow one request from prompt to silicon, and it passes through 11 distinct layers.
- Observability is not a layer; it is a vertical that runs down the side of all 11.
- You do not need to own every layer. You need to know where your line is; ours sits just above the inference engine.
The problem with the term
One phrase, three different buildings
Put three smart people in an architecture review and say inference stack. The GPU engineer hears kernels, KV caches, and batching schedules. The application engineer hears gateways, routers, and agent loops. The person paying the bill hears exactly one thing: cost per token. Same two words, three entirely different buildings, and the meeting goes in circles because nobody notices they are not in the same one.
There is a simple way to end the argument: follow one request. Watch what happens in the seconds after a user hits enter. Every component that request passes through on its way from keyboard to silicon and back is a layer. Trace it honestly, skipping nothing, and you count 11.
The layers
Layer by layer, following one request
Above the inference line · 01-07Application / Client
Where a human hits enter. The only layer users ever see, which makes it ruthless: nobody praises your scheduler, but everyone notices the four seconds a blank screen took. Every problem in the ten layers below eventually surfaces here as one of two feelings: this is slow, or this is wrong.
examples: jusCode · jusInfer · Claude · ChatGPTthe one layer you always have: it is the other ten that arrive over time.
AI Gateway
Every model call in the org walks through one door, or through fifteen doors that you find out about at invoice time. Auth, rate limits, quotas, spend tracking, safety filters, audit logs in a single chokepoint. Its best feature is where requests die: here, for free, instead of three layers down on a GPU, expensively.
examples: jusGateway · LiteLLM (open source)you need it when: three teams hold three different API keys and nobody can answer what we spent last month, and on what.
AI Router
Not every question deserves a frontier model. The router reads each request and picks the right tool: cheap-and-fast for reformat this JSON, frontier for find the race condition, a fallback provider the moment someone's API starts timing out. Before this layer, every request costs the same. After it, cost is a decision you make per request.
examples: OpenRouter · RouteLLM (open source)you need it when: most of your traffic could run on a model 10x cheaper, and it is all going to the expensive one.
Orchestrator
One user click, fifty model calls. Agent loops, tool calls, retries, models chained into pipelines: the orchestrator owns that graph, and its failure modes. What happens when call 37 of 50 times out? If the answer is start over from call 1, this layer is missing and your users pay for it twice.
examples: LangGraph · Temporalyou need it when: a task fails halfway through and nobody can tell you which step died, or why.
Gateway decides who may call. Router decides which model answers. Orchestrator decides what happens across steps. Three layers, three different questions. Collapse them into one box and your diagram lies to you.
Context Layer
The model never sees your application. It sees one string, assembled right here: system prompt, retrieved documents, memory, recent turns, trimmed to fit a window. This layer caps your ceiling. Feed it garbage context and you get confident garbage back, billed per token either way.
examples: RAG pipelines · vector stores · memory systemsyou need it when: answers keep degrading and everyone keeps blaming the model.
Cache Engine
The cheapest token is the one you never generate. Semantic caches answer repeat questions without waking a GPU; prefix and KV caching recycle computation within and across requests. When the same question arrives ten thousand times a day, this layer makes 9,999 of them nearly free, which is why it is routinely the highest-ROI box in the diagram.
examples: GPTCache · Redis semantic cache · prefix/KV cachingyou need it when: your top 100 prompts are near-identical and you pay full price for every one.
Serving Engine
The last layer that thinks in requests: everything below thinks in tokens and tensors. Queueing, admission control, replica placement, autoscaling: the shop-floor schedule deciding which GPU runs which job, and when. Get it wrong and the symptom is sneaky: everything works, just slower every week.
examples: Ray Serve · KServeyou need it when: traffic spikes and your p99 latency quietly triples while dashboards stay green.
Inference Engine
Where the forward pass actually runs. Continuous batching, paged KV-cache management, speculative decoding: some of the finest systems engineering of the decade lives here. It is also, increasingly, a solved problem you download rather than a problem you solve.
examples: vLLM · TensorRT-LLM · SGLangreality check: your competitors run the exact same engine. Nobody wins here anymore; they just avoid losing.
AI Models
The weights. Base models, quantized variants, fine-tunes, LoRA adapters: astonishing artifacts, brutally expensive to create. The quiet irony of the whole stack: the layer everyone talks about most is the one the router above swaps most casually. From layer 03's point of view, models are interchangeable parts.
examples: open-weight & frontier models · LoRA adaptersreality check: a better model ships every month whether you do anything or not. Plan for swapping, not marrying.
Compute Runtime
Kernels, graph compilers, drivers: the translation layer between tensor math and one specific accelerator's instruction set. You will probably never touch it. You will definitely feel it: it is why the same model runs at different speeds on different clouds.
examples: CUDA · ROCm · Triton kernelsHardware
GPUs, TPUs, HBM, interconnect, power, cooling. The only layer with a lead time measured in quarters and a depreciation schedule measured in years. Every layer above exists for exactly one purpose: keep this one busy.
examples: GPU/TPU fleets · HBM · high-speed interconnectWhere we sit
We build above the inference line.
At Kalmantic, our line is drawn just above layer 08, and it is a choice, not an accident. Everything below it is commodity power: extraordinary, improving monthly, and available to us and our competitors on identical terms. Nobody differentiates on a downloaded engine. The seven layers above the line are the harness, where requests are governed, routed, orchestrated, cached, and served, and where a product's cost, speed, and reliability are actually decided. We rent the power. We build the harness.
Why this matters
Building an AI factory? This is your floor plan.
The factory metaphor stuck because it is literal. Raw material in: data, prompts, electricity. Product out: tokens, decisions, actions. And like every factory since the industrial revolution, yours lives or dies on four numbers. Each of the 11 layers exists to move at least one of them:
The factory framing also makes the layers intuitive:
- Gateway (02)the security gate: nothing enters the plant unaccounted for
- Router (03)dispatch: sends each job to the right production line
- Orchestrator (04)the line supervisor: sequences multi-step jobs, handles rework
- Cache (06)the finished-goods warehouse: do not remanufacture what is on the shelf
- Serving (07)the shop-floor schedule: which machine runs which job, when
- Engine + hardware (08-11)the machines themselves: leased power, not your differentiation
Two consequences worth taping to the wall. First, most teams over-invest below the line and under-invest above it. They spend three weeks benchmarking inference engines for a 12% throughput gain while every request re-sends an untrimmed context past a cache that does not exist, leaving 10x on the table two layers up. Second, your line is a strategy, not a default. Decide which layers you own, which you buy, and which you deliberately ignore. Then instrument all eleven anyway, because the observability vertical is the one thing on this diagram you can never outsource.
References
The papers behind the lower layers
- Yu et al., 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 22). Introduced iteration-level scheduling, the continuous batching that defines layer 08. usenix.org
- Kwon et al., 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 23). The vLLM paper: why KV-cache memory, not compute, bounds serving throughput. arXiv:2309.06180
- Leviathan et al., 2023. Fast Inference from Transformers via Speculative Decoding (ICML 23). How small draft models accelerate large ones without changing outputs. arXiv:2211.17192
Test yourself
1. Which layer decides which model serves a given request?
2. Continuous batching and paged KV-cache management live in which layer?
3. Which of these is NOT one of the 11 layers?
FAQ
- Do I need all 11 layers on day one?
- No. A prototype is layer 01 calling a managed API, two layers, and it is fine. The stack fills in as scale forces it: the gateway with the second team, the router with the second model, the cache with the first scary invoice. The mistake is not missing layers; it is not knowing which one your current pain lives in.
- Isn't observability a layer?
- It is a vertical. Tracing, evals, and cost attribution have to span all 11 layers to be useful. A trace that stops at the gateway cannot tell you whether latency came from a cold cache, a queued scheduler, or a slow engine. Draw it down the side, not in the stack.
- Where do agents fit?
- Agents are an orchestrator (04) pattern that leans hard on the context layer (05) and multiplies load on everything below. One agent task can mean fifty model calls, which is exactly why routing, caching, and scheduling stop being optional once agents ship.
- Why draw the line above the inference engine?
- Because that is where differentiation ends today. Engines, weights, and silicon improve on their own schedule and are equally available to you and your competitors. The seven layers above are where your product's cost, latency, and reliability are decided. That is where we choose to build.