Chapter 5 · Philip Kiely, Inference Engineering

Techniques

Five families of techniques make inference faster and cheaper. Each one attacks a specific bottleneck, each has a situation where it pays off, and each has a cost. Only quantization can change output quality, and the others are lossless.

1. 5.1Lower precision speeds prefill and decode by about 30 to 50 percent per step. Test quality before shipping.
2. 5.2Draft several tokens, verify them in one pass. It helps decode latency at low batch sizes.
3. 5.3Reuse the KV cache across requests, and put changing text late in the prompt.
4. 5.4Use tensor parallelism for latency in a node, expert parallelism for MoE throughput.
5. 5.5Split prefill from decode only at large scale, with big models and prefill heavy traffic.

Before 5.1

How to think about techniques

The principle from earlier still holds: the more constraints you add, the better the performance. A second one joins it. The more traffic you have, the more optimization you can afford. Wide parallelism, KV aware routing and dynamic disaggregation only make sense with many GPUs serving one model, often across nodes.

Tuning is never finished, because real traffic keeps changing. Finding the right mix takes patient experiments, and the author recalls an engineer who tried 77 configurations with a script before finding one that doubled tokens per second. Techniques can help each other, as KV cache quantization eases the disaggregation bottleneck, or conflict, as bigger batches take compute away from speculation. The goal is a balanced set that beats the sum of its parts.

Section 5.1

Quantization

Models are trained in a native format, usually BF16 or FP16. Post-training quantization converts weights and other values to a lower precision. Halving precision speeds both phases: prefill runs on Tensor Cores with twice the FLOPS, and decode loads half as many bytes per value. Overhead means about 30 to 50 percent gain per precision step for LLMs, not a full double.

The risk is that precision errors compound. The book shows pi squared and cubed at three precisions, and the errors grow with every operation.

5.1.1 Number formats

The useful inference formats are 16, 8 and 4 bit. A format has a precision, a type (integer or floating point) and a scale factor that maps back to higher precision. Floating point wins for inference because its exponent gives a wider dynamic range, which represents the outlier values that matter. Granularity is how many values share one scale factor: per tensor, per channel or per block. Finer granularity protects outliers but costs storage and compute. MXFP8 and MXFP4 use a scale per 32 values, and NVIDIA's NVFP4 uses blocks of 16 plus a global scale. Integer and aggressive dynamic formats suit local inference, for example GGUF, but production should stick to floating point. FP8 and MXFP8 are the sweet spot.

5.1.2 Approaches

Larger models are less sensitive. Quantization can happen during training, as with GPT-OSS in MXFP4, or after, which is all you can do with finished open weights. NVIDIA TensorRT Model Optimizer is a leading tool and works with all three engines. Components differ in sensitivity, from least to most:

5.1.3 Measuring quality

The bar is no perceptible quality loss, meaning a difference that looks like run to run noise. Use three checks: perplexity, standard benchmarks like MMLU or SWE-bench, and your own evals. Quantization is a scale. FP8 instead of FP4, or weights only, trades less for less risk.

Section 5.2

Speculative decoding

Decode leaves compute idle while it waits for weights. Speculation uses that spare compute. A speculator drafts tokens, the target model verifies them in one pass, accepts the valid ones and adds one token of its own, producing N plus 1 tokens per pass where N is the accepted count. Verification is cheap, like checking a finished sudoku. It improves tokens per second but not time to first token.

Gains depend on draft token cost, draft length and the acceptance rate. Acceptance is high early in a draft and falls later, and one rejected token discards all that follow, so short high-confidence drafts win. Higher temperature lowers acceptance. At high batch sizes compute is saturated and speculation must switch off.

Section 5.3

Caching

Every engine caches the KV within a request. The bigger win is reusing it across requests.

5.3.1 Prefix caching

If two prompts share a prefix such as "Weather in", the second skips prefill for those tokens. This is why APIs charge less for cache hit input tokens. It pays off with long system prompts, code context, retrieved documents and multi-turn chats. The prefix ends at the first token that differs, so any change early in the prompt wipes out the savings. Put novel tokens as late as possible. Reuse from the middle of prompts is active research, for example CacheBlend and LMCache.

5.3.2 Where to store the KV cache

The cache fills fast. Engines let you set its share of memory, for example 80 percent of the VRAM left after weights, which is 64 GB on a B200 with 100 GB used. When it overflows, offload to slower tiers, in descending speed. Dynamo's KV Block Manager moves blocks between levels, keeping hot blocks near the GPU.

5.3.3 Cache-aware routing

With many replicas, route a returning conversation to the replica that already holds its cache, instead of only balancing load. A global cache on networked storage lets any replica reach any cached sequence and survives autoscaling, though a replica with a hot GPU cache still answers faster.

5.3.4 Long context

A context becomes long when its KV cache causes problems, often past 32K, 64K or 128K tokens. Attention can then become the main consumer of VRAM. General remedies are FlashAttention, PagedAttention and chunked prefill, which splits long inputs into pieces run alongside decode. Benchmark with very long inputs. If one GPU is still not enough, parallelize.

Section 5.4

Model parallelism

Frontier models do not fit on one GPU. In FP8 each billion parameters needs about a gigabyte, so DeepSeek-V3.1 at 671B overflows a single B200. Even four B200s with 720 GB hold the weights but leave no room for the KV cache, so a full node of eight is needed. The limit on scaling is communication between GPUs, which is far slower than VRAM, hence topology aware parallelism.

5.4.1 Tensor parallelism

The default for multi-GPU inference. Each layer is split across GPUs, and results are combined with an all-reduce before the next layer. Inside an NVLink node that cost is small, and raising TP improves per-user tokens per second for large models.

5.4.2 Expert parallelism

Experts are divided across GPUs, so with 128 experts on EP8 each GPU hosts 16. Each token takes as long, but the system handles more tokens at once. It needs less communication than TP, since the small router is replicated, so it scales across nodes. Many deployments mix TP and EP.

5.4.3 Multi-node inference

Over InfiniBand, TP is too chatty. For dense models use TP inside each node and PP between nodes, such as TP8PP2. For MoE you can also use EP16. TP8PP2 gives lower latency per user and EP16 higher total throughput. Unless the model and cache force you across nodes, extra hardware is better spent on replicas or disaggregation.

Section 5.5

Disaggregation

Prefill is compute bound and sets TTFT, and decode is memory bound and sets TPS. Under heavy load on one node they compete. Disaggregation puts them on separate engines: the prefill engine builds the KV cache and the first token, ships the cache over the interconnect, and the decode engine produces the rest. Each engine can be tuned separately, for example prefill uses lower TP than decode. With conditional disaggregation the decode engine first checks whether the prompt is short or already cached and prefills locally if so, which suits real traffic better.

5.5.2 When to use it

It needs many GPUs and engineering work, so use it only when three things hold. Check your own situation:

5.5.3 Dynamic disaggregation with Dynamo

Dynamo adds a prefill queue, conditional routing on thresholds, and NIXL based KV transfer that can transpose cache layouts when TP differs. Engine counts are written xPyD, so 5P3D means five prefill and three decode engines, adjustable at runtime. New bottlenecks appear: the prefill queue and decode KV capacity. Tune the local prefill threshold and xPyD, and add KV quantization or offloading for capacity.

Summary

Chapter map

The whole chapter on one page. Each branch is a section, and each leaf is the idea to remember from it.

Retention

Recall check

Answer in your head first, then select the card. Coming back to these in a week works better than rereading the chapter.