Inference Is a New Problem
Training is a batch job: hours of heavy computation, and no one is waiting on a single answer. Inference is a live service: thousands of requests per second, each with a latency budget measured in hundreds of milliseconds. The optimization targets are completely different — you’re no longer maximizing accuracy, you’re maximizing requests-per-second within a latency SLA.
Latency vs Throughput
- Latency — time from request to first/full token (your users feel this).
- Throughput — requests or tokens served per second (your costs depend on this).
They trade off. Small batches minimize latency; large batches maximize throughput. Production systems tune batching to fill the gap between the latency budget and GPU idle time.
Batching
A GPU can process many sequences in parallel. Dynamic batching groups requests that arrive close together into one forward pass. An LLM serving stack also reuses the KV cache: the key/value tensors of previously generated tokens, cached so the next token only recomputes the newest position. Without the KV cache, every token would re-process the whole prompt — the single biggest serving optimization.
Quantization
Model weights are typically 16-bit floats. Quantization shrinks them to 8-bit or 4-bit integers:
| Precision | Memory | Speed | Quality impact |
|---|---|---|---|
| FP16/BF16 | Baseline | Baseline | None |
| INT8 | ~½ | Faster | Negligible |
| INT4 | ~¼ | Fastest | Small, task-dependent |
Smaller weights mean more of the model fits in GPU memory, higher batch sizes, and cheaper serving — at the cost of a small quality risk that must be evaluated per task.
GPU vs CPU Inference
| CPU | GPU | |
|---|---|---|
| Cost per token | Higher | Lower at scale |
| Latency per request | Higher | Lower |
| Best for | Small models, cold-start-tolerant, sporadic load | Large models, steady high load |
Small models (embedders, classifiers, rerankers) often serve fine on CPU. Large generative models need GPU. The wrong choice burns money or latency.
Autoscaling and Cold Starts
Model endpoints scale on request volume, but loading a multi-gigabyte model takes tens of seconds — the cold start is real. Autoscaling policies must provision ahead of spikes, keep a warm minimum, and accept that new instances can’t serve instantly. Caching repeated completions and reusing pooled GPU instances are standard mitigations.
When It’s the Right Tool
Choose self-hosted serving when you control the stack and traffic is steady; choose a managed inference API when traffic is spiky, you want zero ops, or the model changes often. The right answer depends on cost-per-token, latency budget, data-residency requirements, and engineering time — not on fashion.
Practice Trajectory
- Explain why KV-cache reuse is the biggest LLM serving win.
- Estimate the memory saved by INT8 vs FP16 for a 7B-parameter model.
- Describe a load pattern where CPU inference is the better call.
- Design an autoscaling policy that survives a cold start.
- Compare self-hosted vs managed inference for two scenarios you can name.