Chapter 0 · Philip Kiely, Inference Engineering

Inference

Training teaches a model. Inference is everything that happens after, when real users and real traffic hit the weights. For generative AI that second phase has become an engineering discipline of its own, built from three layers: runtime, infrastructure and tooling.

Fig. A · Where inference sits · synthesis · select a stage

  1. PhaseTraining learns the weights once. Inference serves them continuously, and it is where the product meets the world.
  2. KindsClassic ML inference is cheap and simple. Generative inference needs real systems work.
  3. LayersRuntime makes one GPU fast, infrastructure scales it out, tooling keeps humans productive.
  4. BookThe rest of the book walks up the stack, from models and hardware to production.

Context

Classic versus generative inference

Inference has always existed. A fraud model scoring a payment or a gradient boosted tree ranking search results is inference too. Those models are small, run happily on CPUs, and rarely need more than a container and a load balancer.

Generative models changed the shape of the problem. They are large, they need GPUs, and each request does a lot of work, so the gap between a naive deployment and a tuned one is measured in multiples of cost and latency.

Fig. B · Effort by kind of model · illustrative · select a kind

The stack

Runtime, infrastructure, tooling

The book organizes inference engineering into three layers. Each one answers a different question, and each one fails in a different way. Select a layer to see what lives in it.

Fig. C · The three layers · synthesis

How infrastructure grows

Infrastructure work arrives in a predictable order. Autoscaling comes first. Past a few hundred GPUs you need capacity in several regions or clouds. Those pools start as silos, and the end state is a single global pool that routes each request to the best place, which also helps with reliability and distance to users.

Fig. D · Infrastructure growth path · synthesis · press play

Orientation

How the book is laid out

Each following chapter climbs one rung. Select a chapter to see its job.

Fig. E · Book structure · synthesis

Summary

Chapter map

The whole chapter on one page. Each branch is a section, and each leaf is the idea to remember from it.

Fig. M · Chapter 0 at a glance · synthesis

Retention

Recall check

Answer in your head first, then select the card. Coming back to these in a week works better than rereading the chapter.