Chapter 1 · Philip Kiely, Inference Engineering

Prerequisites

Before you make anything fast, decide what fast means for your product. Optimizing is a series of trade-offs among latency, throughput and quality, so the first job is knowing which side of each trade your product sits on.

The chapter opens with an athlete analogy. A professional football player is not as big as a sumo wrestler, as fast as an Olympic sprinter, or as strong as a champion powerlifter. The body is tuned for one position over a full season. An inference system should be tuned the same way, for your model, your product and your traffic. The more constraints you can introduce, the better the outcome you can reach.

1. 1.1Decide between shared and dedicated inference, and move only when scale, specialization or orchestration demands it.
2. 1.2Let your app decide the targets: online or offline, consumer or business, and which compliance rules apply.
3. 1.3Choose the smallest model that passes your own evals, then consider fine-tuning or distillation.
4. 1.4Measure with TTFT, TPS and percentiles, and separate inference time from end-to-end time.

Before 1.1

Know your five inputs

Optimization means picking the best option from a range of options, and you cannot pick until you know what best means. The chapter gives five things to know about your product. Select a node to read the question behind it.

Early in a product's life these answers are often unclear. In that phase it is usually better to use off-the-shelf APIs than to invest in dedicated inference. As the product scales, the requirements firm up and inference engineering becomes worth the effort.

1.1

Scale and specialization

There are two ways to add a model to a product. With shared inference you send traffic to a public API endpoint and pay per million tokens or a similar consumption metric. With a dedicated deployment you rent GPUs, or buy them, run a service just for your application, and pay for GPU time.

This is not the same as closed versus open models. Plenty of shared endpoints serve open models, and many labs sell dedicated setups for their closed models. The link is that open weights unlock unrestricted dedicated inference.

Most AI products start on pay-per-token APIs because the trade-offs make sense while you look for product-market fit. Over time, moving to dedicated deployments becomes essential for three reasons.

The switch makes you responsible for your own inference engineering. That buys flexibility and control, and it also widens your engineering surface and raises the floor of your monthly spend. Switch only when there is a clear and immediate business need.

1.2

About your app

Every inference decision you make is downstream of your use case. A basketball coach and a gymnastics coach recruiting at the same school want different students, because the sport decides what to look for. How your inference system will be used decides how you build it.

Two kinds of builders have to work at the highest level of generality. Foundation model labs sell consumption through a public API and must support many usage patterns. Inference platforms, internal or commercial, must support any model and any use case. Most AI-native applications are vertical apps such as code editors or customer service agents, and for those the advice reverses: add as many constraints as you can.

1.2.1AI-native applications

Generative models have opened new categories of products, and each category leans on different models and modalities. The chapter offers a small sample to show how wide the range of considerations is.

1.2.2Online versus offline

One of the main trade-offs in inference engineering is latency against throughput. Lower latency makes the application faster. Higher throughput makes it cheaper at scale, because fewer GPUs serve the same number of users. Code completion, chat and voice agents are online workloads with an impatient person waiting, so they optimize for latency.

Offline batch jobs run on high-throughput deployments. Each request would feel too slow to a person, but the system finishes far more requests per hour in parallel. Examples are transcribing a back catalog of audio, embedding or analyzing documents on a schedule, and preparing large corpora for model training.

A single model can serve both. Whisper, a speech-to-text model, can power a real-time dictation app and a batch transcription job. With enough volume on both, it is more cost effective to run two deployments of the same model, one tuned for latency and one tuned for throughput.

1.2.3Consumer versus B2B

Consumer apps are more cost-sensitive and less predictable, and a launch or a marketing campaign can spike usage overnight. Business products tend to have better margins and steadier usage, but they sit in a revenue path and are held to a higher standard of reliability.

In both cases compliance can limit your infrastructure options, especially in regulated industries. Work closely with security and legal experts to make sure what you operate is compliant.

1.3

Model selection

All else being equal, including hardware, runtime, optimizations and architecture, a smaller model with fewer parameters is faster and cheaper than a larger one. So the most important performance decision is which model you work with, not which runtime or speculation algorithm you pick.

Your choice also limits which optimizations are available, because inference engines differ in how well they support each architecture. Stick with popular architectures to get robust support across the tooling landscape.

Before product-market fit, it is not worth spending time or money on your own inference. When it is time to scale, the advice flips. Find or create the smallest, easiest-to-run model that is smart enough for the task. Sometimes that is still a trillion-parameter frontier model, but it is always worth checking whether a smaller, cheaper, faster one can do the job.

1.3.1Model evaluation

Evals are the practice of systematically measuring model intelligence, and they are a prerequisite for inference engineering. They help in two ways. They make you spend time wisely, because you confirm a model is useful before investing in making it fast. They also establish a baseline, because some optimizations reduce quality and you need something to compare against.

Evals differ from standard intelligence benchmarks such as MMLU or SWE-bench. Benchmarks measure capability on common tasks, while evals are tailored to your product, domain and tasks. Benchmarks are useful for shortlisting models, but they saturate and get gamed. Goodhart's Law says that when a measure becomes a target, it stops being a good measure, and frontier labs have a strong incentive to post record benchmark scores. Elo rating from head-to-head win rates gives a better read on overall intelligence, but nothing replaces measuring how a model performs in your application.

1.3.2Fine-tuning for domain-specific quality

Fine-tuning takes a pre-trained foundation model and adapts it to a specific use case by introducing new data. The model's weights change while its overall architecture stays the same. If you can fine-tune a small model to pass your evals, hitting latency and cost targets gets much easier.

Text-to-SQL is a strong example. General-purpose coding models are good at SQL but have hundreds of billions of parameters. SQL is a constrained language, so an application that only turns natural-language prompts into queries can reach equivalent performance with a tiny fine-tuned model of a few billion parameters. Most domains will not shrink that far. This works when you have three things.

1.3.3Distillation

Distillation keeps most of a large model's intelligence at a fraction of the size. A large teacher model trains a smaller student to emulate its behavior. Unlike fine-tuning on input-output pairs, the student sees the teacher's actual probability distributions and not only its final answers. Fine-tuning teaches a model to do better in a domain. Distillation teaches it to copy the behavior of a larger model, good and bad.

Distillation sees substantially less real-world use than fine-tuning. When a frontier lab releases a family of models in different sizes, the small ones are usually trained independently, so that the biases of the large model do not cap the small ones. Distillation matters mainly when a lab only trains a large model and wants to make it accessible.

The best-known case is DeepSeek-R1, released in January 2025 at 671B parameters. DeepSeek also released distilled versions built on the most popular architectures of the time, Llama 3 and Qwen 2.5. They showed similar reasoning behavior with lower benchmark scores, and because they were small and used familiar architectures, they could reuse existing performance work.

1.4

Measuring latency and throughput

The two most common metrics for LLMs are TTFT, time to first token, and TPS, tokens per second. With streaming output, TTFT asks how long a user waits to see the first token. TPS asks how many tokens arrive each second after that. TTFT is driven by compute-bound prefill, and TPS is driven by bandwidth-bound decode. Lower TTFT and higher TPS both mean better latency.

TTFT is a clear term, but TPS is less precise. It can mean tokens per second per user, which is a latency metric, or tokens per second for the whole service, which is a throughput metric. Most people mean the per-user kind. When it matters, use the specific terms.

TTFT and TPS suit user-facing systems that stream output, such as chatbots. For something like an agent's tool call, the tokens are not useful one by one, so you measure total response time instead.

1.4.1Latency percentiles

The naive approach is to look at average TTFT or TPS, which hides the shape of the distribution. Total LLM response time is right-skewed. Most requests cluster near a mode, and outliers take far longer. If one request in ten takes several seconds, the product does not feel snappy, so engineers measure percentiles.

Driving down the average still matters, but good performance work also reduces P90 and P99 latency, which gives users a more reliable experience.

1.4.2End-to-end metrics

The other distinction is whether you measure only inference time, the on-GPU time to generate tokens, or the end-to-end time that includes network latency and queueing. Both are useful. Inference time tells you how effective your model performance work is. End-to-end time shows what your users perceive. When inference is fast but end-to-end is slow, turn to infrastructure instead of model optimization.

Summary

Chapter map

The whole chapter on one page. Each branch is a section, and each leaf is the idea to remember from it.

Retention

Recall check

Answer in your head first, then select the card. Coming back to these in a week works better than rereading the chapter.