How to design an AI chat assistant like ChatGPT, JusInfer.Chat and others
The interview question is really two serving-layer patterns: stream tokens in real time, and manage long-running work on scarce GPUs.
7/1/2026 · 4 min · jusCode · Read as Markdown
TL;DR
Designing an AI chat assistant is two serving-layer patterns: stream tokens in real time over SSE and optimize time-to-first-token, and manage long-running generation on scarce GPUs with a queue and scheduler. Treat the model as a black box; persist and summarize context so cost stays flat.
- Treat the LLM as a black box; you are designing the serving system, not the model.
- Optimize time-to-first-token and stream tokens over SSE; a blank screen feels broken.
- GPUs are scarce: queue requests and let a scheduler decide who gets compute, and when.
- Enforce fairness with rate limits and tiered priority so heavy users cannot starve everyone else.
- Persist every turn for resumable chats, and summarize old context so cost stays flat.
Demo · Why streaming is the product
The reframe
The model is the easy part.
You are not designing an LLM. Treat the model as an opaque service you call: no training pipelines, no internals. Every interesting decision lives in the layer around it: how tokens get back to the user fast, how requests are routed onto expensive GPU workers, how conversations persist and resume, and how cost stays sane as context grows. Scope it to text in, text out, at roughly 200M daily active users.
Step 1 · Scope
Requirements draw the line
Strong candidates decide what is in and what is out, then name the constraints that shape the architecture.
Functional in scope
- Send a prompt in a chat and receive an AI-generated response.
- View past chats and resume one, with prior context carried into the next prompt.
Deliberately cut out of scope
- Editing or branching existing messages.
- Images, audio, or video: text only.
- Sharing chats, tool calling, browsing.
- Full-text search over chat history.
Step 2 · High-level design
One request path, one long-running path
A synchronous path for chat history, an asynchronous path for generation, with tokens streamed back the moment they exist.
Step 3 · Name the patterns
Two patterns carry the whole design
Recognize them, and the architecture assembles itself.
Real-time updates
Generation takes seconds, but the product must feel instant. Push each token to the client the moment the model produces it. Perceived latency collapses to the time-to-first-token.
why not WebSockets: one-way flow, simpler infra
UX: smooth the stream client-side, buffer bursts
metric: TTFT, not total completion time
Managing long-running tasks
A request can run for many seconds on hardware you cannot just add more of. Decouple accepting the request from executing it: enqueue jobs, schedule GPU workers, support cancellation.
queue: buffers spikes, protects the fleet
scheduler: routes by load, tier, and model
cancel: user hits stop → free the GPU immediately
Step 4 · Deep dives
Where interviewers push
How do tokens get back fast, and smoothly?
Hold the connection open and flush tokens as they arrive rather than buffering the whole response.
Answer shape: Stream over SSE end to end (worker → service → client), avoid proxies that buffer, and render at a steady cadence even when tokens arrive in bursts. Write the completed message to the database after the stream ends, so a slow write never blocks the user.
How are requests routed across GPU workers?
Workers are stateful while generating and wildly expensive, so naive round-robin wastes them.
Answer shape: A scheduler tracks worker load and in-flight generations, batches compatible requests, and admits new work only when capacity exists. Under overload, requests queue rather than crush the fleet.
How do you keep heavy users from monopolizing GPUs?
Compute is the product's real currency, and paid tiers expect a visibly better experience.
Answer shape: Per-user rate limits plus tiered quality of service: separate queues or priority weights so premium traffic schedules first, while free-tier degrades gracefully instead of failing.
How do you control cost as conversations grow?
Every turn re-sends prior context, so a long chat makes each prompt bigger and pricier.
Answer shape: Cap the context window deliberately: keep recent turns verbatim, compress older ones into a running summary, and drop what no longer matters.
Test yourself
1. Which latency metric matters most for a chat assistant?
2. Best default transport for one-way token streaming?
3. A conversation is 200 turns long. How do you keep inference cost flat?
FAQ
- Why SSE instead of WebSockets?
- Token streaming is one-way, server to client. SSE rides on plain HTTP, reconnects automatically, and needs no special infrastructure. WebSockets earn their complexity only when the client must also push data mid-stream.
- Does this design apply beyond chat assistants?
- Yes. Any product that calls an LLM, coding agents included, inherits the same two problems: making generation feel instant, and rationing scarce inference capacity. The serving-layer patterns transfer directly.
- Where should the response be written to the database?
- After the stream completes, not during it. Persisting per token multiplies writes for no user benefit; persisting at the end keeps the hot path free. Handle a mid-stream crash by marking the message incomplete and letting the client retry.