Chapter 4 · Philip Kiely, Inference Engineering

Software

Between the GPU and your users sits a stack of software. Most inference work happens near the top, configuring engines and orchestrating GPUs, but a good mental model of each layer below tells you where a problem really lives.

Fig. A · The software stack · synthesis · select a layer

  1. 4.1CUDA kernels do the work. Most engineers select kernels and rely on fusion instead of writing them.
  2. 4.2PyTorch is the standard framework, safetensors is the weight format, and compilation targets a GPU.
  3. 4.3Choose between vLLM, SGLang and TensorRT-LLM on a deployment by deployment basis.
  4. 4.4Dynamo orchestrates engines at scale with KV aware routing and disaggregation.
  5. 4.5Benchmark with realistic traffic, change one thing at a time, and profile to learn why.

Section 4.1

CUDA

CUDA is NVIDIA's platform and programming model for running parallel work on its GPUs. It is not a language: you write C++, and a compiler such as nvcc produces separate CPU and GPU code. Its parts are the kernel (a function that runs in parallel on the GPU), the graph (a DAG of kernels for repeated workflows), the driver and the runtime API. A kernel can replace thousands of sequential loop iterations with thousands of parallel threads.

4.1.1 Kernels for inference

You rarely start from nothing. Libraries supply the building blocks.

Fig. B · Where kernels come from · select a library

4.1.2 Kernel selection

Kernels are tied to hardware details, such as bandwidth or Tensor Core layout, so a kernel tuned for an H100 may not suit a B200 and the reverse may not run on Hopper at all. Most selection is automatic in PyTorch and TensorRT-LLM, but engineers sometimes choose by hand. DeepSeek released DeepGEMM, FP8 matrix multiply kernels for Hopper, and you can plug them in as a plugin. Check compatibility before changing GPUs.

4.1.3 Kernel fusion

Running two kernels back to back writes the first result to memory and reads it straight back. Fusing them into one kernel skips that round trip, which matters most in bandwidth bound decode. Compilers find simple fusions, while complex ones like FlashAttention are handwritten and loaded as plugins. A common pattern is fusing a matrix multiply, bias add and activation.

Fig. C · Kernel fusion · using the book's multiply by 2 then 3 example · choose a version

Section 4.2

Deep learning frameworks and libraries

Frameworks bridge raw CUDA and ready-made engines. PyTorch has become the clear leader. TensorFlow was prominent in the 2010s but has faded, and JAX is a simpler research project with sharp edges.

Fig. B · The framework layer · select a topic

The industry is splitting between the control of handwritten PyTorch and the convenience of prebuilt engines. For supported models it is increasingly common to skip the intermediate representation and export safetensors weights straight to vLLM or TensorRT-LLM.

Section 4.3

Inference engines

Three engines are competitive: vLLM, SGLang and TensorRT-LLM. They give strong out-of-the-box performance for LLMs and similar architectures, ship continuous batching, and support the main techniques (quantization, speculation, prefix caching, parallelism, disaggregation). vLLM and SGLang are general and easy with day-zero model support, while TensorRT-LLM is harder but usually fastest. The author uses all three, most often TensorRT-LLM.

Fig. B · Engine comparison · qualitative, from the book's table · select an engine

Fig. C · When to use each · select an engine

Section 4.4

NVIDIA Dynamo

Dynamo, announced in March 2025, is a distributed serving layer that uses vLLM, SGLang or TensorRT-LLM as backends. It provides KV cache re-use with prefix aware routing, disaggregation of prefill and decode with independent scaling, and multi-node parallelism, often with expert parallelism. A planner can scale prefill and decode workers to meet TTFT and TPS targets.

Fig. B · A Dynamo deployment · synthesis · select a part or press play

More scale unlocks more techniques. Dynamo suits big models and heavy traffic such as trillion parameter models with many concurrent users. Without enough volume for disaggregation and KV aware routing to matter, it is extra complexity and overhead, and an engine alone is enough.

Section 4.5

Performance benchmarking and load testing

Without precise benchmarks you cannot tell whether an optimization worked. The best benchmark shadows real production traffic, copying requests to the test system without affecting the originals. Otherwise simulate traffic across several dimensions.

Fig. B · What a realistic benchmark matches · select a dimension

4.5.1 and 4.5.2 Tooling and tips

Common tools are SGLang Genai-bench, NVIDIA GenAI-Perf and the general purpose load tester Locust. Open evals such as MMLU, gsm8k, SWE-bench or HumanEval double as realistic inputs and as a spot check that quality did not drop. Start from a baseline, keep the configuration constant, run several times, change one variable at a time, and test each optimization alone and together, since some conflict, such as speculation at large batch sizes.

Fig. C · The optimization loop · synthesis · select a step

4.5.3 Profiling

A benchmark gives a figure such as P90 TTFT of 350 ms. A profiler shows where each millisecond went. Benchmarking says how the system performs, profiling says why. Most engineers who configure TensorRT-LLM never profile, but contributors to engines, authors of custom PyTorch services and people working on video generation should. Tools are PyTorch Profiler, NVIDIA Nsight Systems for system wide traces and Nsight Compute for single kernels.

Summary

Chapter map

The whole chapter on one page. Each branch is a section, and each leaf is the idea to remember from it.

Fig. M · Chapter 4 at a glance · synthesis

Retention

Recall check

Answer in your head first, then select the card. Coming back to these in a week works better than rereading the chapter.