Chapter 2 · Philip Kiely, Inference Engineering

Models

To make inference fast you have to know what the model is actually doing. This chapter opens up two families, autoregressive language models and iterative image generators, then uses simple arithmetic to find which resource limits each one.

1. 2.1A network is layers of matrix multiplications with activations that bend the result.
2. 2.2An LLM prefills the prompt once, then decodes one token at a time using a KV cache.
3. 2.3Image models start from noise and denoise in 30 to 50 steps, each with two passes.
4. 2.4Compare arithmetic intensity with the GPU ops:byte ratio. Prefill is compute bound, decode is memory bound.
5. 2.5Attention is the costliest part, so kernels like FlashAttention and variants like sliding window matter.

Section 2.1

Neural networks

A neural network is nodes arranged in layers: an input layer, hidden layers and an output layer. The values moving between layers are hidden states, and their width is the dimensionality. Modern LLMs are decoder only, while models like Whisper pair an encoder with a decoder.

2.1.1 Linear layers and matmul

The workhorse is the linear layer, which multiplies the input by a weight matrix and adds a bias. Almost all the weights in a model sit in these layers, which is why inference is dominated by matrix multiplication.

2.1.2 Activation functions

Stacked linear layers collapse into one linear layer, so a nonlinear activation sits between them. ReLU clips negatives, and smoother variants such as SiLU, Swish and SwiGLU are common in LLMs. Move the slider to compare them.

Section 2.2

LLM inference mechanics

Text is split into subword tokens from a vocabulary of over a hundred thousand entries. A request holds an input sequence, possibly a reasoning sequence and an output sequence, all inside the context window, with max_tokens as a cap. A chat template wraps messages into the exact format the model was trained on.

Generation has two phases. Prefill reads the whole prompt in parallel and builds the KV cache. Decode then produces one token per step: the model outputs a logits vector the size of the vocabulary, which is normalized and sampled until a stop token appears.

Temperature reshapes the distribution, top-k keeps only the k likeliest tokens, and top-p keeps the smallest set whose probability adds up to p. Logit biasing nudges specific tokens up or down.

2.2.1 and 2.2.2 Architecture and transformer blocks

The model's config.json lists its sizes, and its name tells you a lot: a name such as Qwen3MoeForCausalLM says it is a mixture of experts model trained for causal language modeling. Instruct variants and fine-tunes (often LoRA) sit on top of a base model. Inside, an embedding layer feeds a stack of identical blocks, each with attention, a feed-forward network and normalization, and a final LM head produces the logits.

2.2.3 Attention

Each token produces a query, a key and a value. Attention scores a query against all earlier keys, scales and normalizes the scores, and takes a weighted sum of the values. Multiple heads do this in parallel. A causal mask stops tokens from seeing the future. Without caching, every step would redo the work for all earlier tokens, which is why the KV cache exists.

2.2.4 Mixture of experts

MoE models replace the feed-forward network with many experts and a router that picks a few per token. Qwen3-235B-A22B has 235B parameters but activates about 22B per token, picking 8 of 128 experts in each of 94 layers. In a large production batch, nearly every expert is touched anyway, so MoE saves compute more than memory unless expert parallelism spreads the experts. MoE tends to appear above roughly 100B parameters, while small models stay dense.

Section 2.3

Image generation inference mechanics

An image pipeline chains a text encoder, a denoising model and a VAE that moves between pixels and a much smaller latent space. Extras such as LoRAs and ControlNets, often wired in with ComfyUI, adjust the result. Generation starts from noise and removes it over 30 to 50 steps, controlled by the prompt, a negative prompt, step count, guidance scale and image size.

Each step runs the denoiser twice, once with the prompt and once without, to apply guidance. So 50 steps means 100 forward passes. Compare that to a few-step model.

2.3.1 Architecture

Modern denoisers are diffusion transformers that look at images as small patches. SDXL used a CLIP based encoder and a base plus refiner. Newer models such as Qwen Image use a 7B language model as the text encoder, a denoiser of about 20B parameters and dual VAE encoding. Research is now blending diffusion with autoregressive LLM architecture, which can output variable sizes and avoid dozens of forward passes.

2.3.2 and 2.3.3 Few-step models and video

Few-step models, built by latent consistency or distillation, finish in eight steps or fewer and run 80 to 90 percent faster at some quality cost. Video models are like image models but bigger: three to five times the parameters and far more latent data. They keep the whole clip in a latent space with a time axis so frames attend to each other, instead of generating frame by frame, which compounds errors. They typically run at batch size one, using a whole eight GPU node for one request.

Section 2.4

Calculating inference bottlenecks

A GPU has two key resources: compute in operations per second and memory bandwidth in bytes per second. A bottleneck is an imbalance where one sits idle while the other is saturated, so optimizing the idle one achieves nothing.

2.4.1 Ops:byte and arithmetic intensity

Divide compute by bandwidth to get the hardware ops:byte ratio. An H100 does about 989 teraFLOPS in FP16 against 3.35 TB/s, a ratio near 295. An algorithm's arithmetic intensity is its operations divided by bytes moved. Below the hardware ratio it is memory bound, above it compute bound, which is what a roofline chart shows.

2.4.2 LLM bottlenecks

Prefill loads the weights once and does large matrix multiplications over the whole prompt, so intensity is high and it is compute bound. It sets time to first token. Decode loads the weights for every single token and does small vector-matrix work, so intensity is low and it is memory bound. It sets tokens per second. The book's worked example, standard attention with d=128 and N=4096 in FP16, comes out near 62, far below 295. Batching raises decode's useful work per byte moved.

2.4.3 Image bottlenecks

Image and video models have about a tenth the parameters of frontier language models, but attention over the full latent is just as demanding, so like prefill they are compute bound.

Section 2.5

Optimizing attention

Attention is among the most expensive parts of inference, and it is delicate because every token depends on every earlier one. There are two routes: better implementations, which are lossless, and new algorithms, which trade some quality for better scaling.

Sliding window attention only looks at the last w tokens, turning N squared into N times w. Training with the window keeps quality high. See how the saving grows with length.

Another direction replaces attention. Mamba style state-space models update a recurrent state with linear scaling, and hybrid models mix such blocks with transformer blocks, as in NVIDIA Nemotron 3 Nano.

Summary

Chapter map

The whole chapter on one page. Each branch is a section, and each leaf is the idea to remember from it.

Retention

Recall check

Answer in your head first, then select the card. Coming back to these in a week works better than rereading the chapter.