Chapter 3 · Philip Kiely, Inference Engineering

Hardware

Inference engineers work a level above the silicon, but a clear mental model of the GPU decides every hardware choice. Two numbers matter most: how much compute the chip has and how fast it can move bytes.

Fig. A · Inside a GPU · synthesis · select a part

3.1Tensor Cores set compute, VRAM size limits the model, bandwidth limits decode.
3.2Pick among Lovelace, Hopper and Blackwell, with Rubin coming. Newer means lower precision and inference features.
3.3An instance bundles GPUs with everything else. Nodes hold eight GPUs linked by NVLink, and InfiniBand links nodes.
3.4Other chips bet on bandwidth, power efficiency or platform integration, and all fight the CUDA moat.
3.5Local inference removes latency and cost but is limited by hardware, heat, fragmentation and battery.

Section 3.1

GPU architecture

3.1.1 Compute

A GPU is built from Streaming Multiprocessors, and each contains three kinds of compute: CUDA cores for scalars, Tensor Cores for vectors and matrices, and special function units for operations such as sin, cos and log. Inference compute is measured in Tensor Core FLOPS, because matrix multiply and accumulate is the foundation of inference. A GPU also runs tens or hundreds of thousands of threads at once.

Spec sheets show dense and sparse FLOPS. Sparse assumes 2:4 structured sparsity and is often double the dense figure, but inference is dense by default, so read the dense column. FLOPS roughly double each time precision halves, so compare chips at the same precision. Compute is the limit for prefill and for image and video generation.

3.1.2 Memory and caches

VRAM is high bandwidth DRAM (HBM3, HBM3e or HBM4). On chip SRAM forms three cache levels: L0 for a Tensor Core, L1 shared per SM, and L2 shared across SMs. An H100 has 256 KB of L1 per SM and 50 MB of L2. VRAM size caps the model you can load, and you want the weights plus at least 50 percent headroom for the KV cache. VRAM bandwidth limits decode at low to medium batch sizes, which is why an H200 beats an H100 for tokens per second.

Fig. B · Will it fit, and how fast · illustrative rule of thumb · choose a model size and GPU

Section 3.2

GPU architecture generations

A GPU name has a letter for the architecture and a number for the model within it. The H100 replaced the A100, the H200 is a bigger Hopper, and the B200 supersedes both. Bigger numbers mean larger, more powerful and more expensive parts. New architectures arrive every one to two years, and most deployments today use Lovelace, Hopper or Blackwell.

Fig. B · Spec comparison · FP8 dense compute, memory and bandwidth, normalized to the largest · select a GPU

The generations

3.2.1 Hopper

Released 2022. Added FP8, which doubles Tensor Core speed and halves bandwidth per value. H100 and H200 are the most widely used inference GPUs and enable FlashAttention 3.

3.2.2 Ada Lovelace

A graphics oriented counterpart to Hopper with FP8 but no NVLink. L4 suits small embedding and vision models. L40 is rarely the right pick, since MIG slices of an H100 beat it.

3.2.3 Blackwell

Released 2024. Adds FP4 and microscaling formats (MXFP8, MXFP4, NVFP4). B200 and B300 are the new gold standard for demanding models and video, with FlashAttention 4.

3.2.4 Rubin

Due in 2026 with HBM4 for decode and a CPX chip built for compute bound prefill. Wait for real benchmarks, as software support takes about a year to mature. Feynman follows in 2028.

3.2.5 Grace and Vera CPUs

ARM CPUs paired with GPUs on superchips. NVLink chip to chip gives up to 900 GB/s, so offloading LoRA weights or KV caches to CPU memory is much faster than over PCIe. Vera replaces Grace with Rubin.

Section 3.3

Instances

The cloud allocates GPUs as instances: a virtual machine with GPUs, host CPUs and memory, storage, networking and the interconnect. Providers build these differently, and any component can be the bottleneck. Even one GPU can differ, since A100 SXM has about 5 percent more memory bandwidth than the PCIe version.

3.3.1 Multi-GPU instances

A node holds eight GPUs. NVLink connects GPUs one to one, up to 1800 GB/s on Blackwell and 900 GB/s on Hopper, and NVSwitch gives all to all connectivity inside the node. Beyond eight GPUs you need InfiniBand, up to 400 Gb/s per network card, still the fastest node to node option, ahead of Ethernet at 100 Gb/s. NVL72 racks link 72 Blackwell GPUs with NVLink for the largest models.

Fig. B · Interconnect bandwidth · GB/s, per link · hover a row

3.3.2 Multi-instance GPUs

Sometimes the GPU is too big for the model. MIG splits an A100, H100, H200 or B200 into up to seven slices, each with its own share of compute, memory and host resources. Seven slices because an H100 has 132 SMs, which does not divide evenly, so seven equal slices are made and a few SMs idle. A small model like a 3B parameter TTS can use two MIG instances more efficiently than a whole GPU.

Fig. C · Slicing an H100 · derived from 132 SMs · choose slices

Section 3.4

Other datacenter accelerators

NVIDIA leads, but many companies build alternatives: AMD MI350, AWS Inferentia and Trainium, Cerebras WSE-3, Etched Sohu, Furiosa RNGD, Google TPU, Groq LPU, Qualcomm Cloud AI 100 Ultra and SambaNova RDU. Each bets on a specific edge.

Fig. B · Where challengers compete · select a bet

Section 3.5

Local inference

Local, edge or on-device inference runs the model on the user's own hardware. It has four advantages: no network latency, independence from connectivity and server load, better privacy, and no GPU bill for the developer. It also has four weaknesses: modest hardware, thermal limits, a fragmented support matrix and battery drain. Design for the median user's device, not the enthusiast's.

3.5.1 Desktop inference

Apple's unified memory trades speed for capacity, which shows in the highest end options. Tools such as Ollama and llama.cpp run aggressively quantized 100B plus models, MoE models help because a request touches few weights, and ComfyUI is popular for image generation.

Fig. B · Capacity versus speed · normalized to the larger value · select a machine

3.5.2 Mobile inference

Mobile makes up most on-device workloads today, using Google's AI Edge SDK and ML Kit with Gemini Nano and Gemma, or Apple's Foundation Models and Core ML. Even high-end phones struggle beyond one or two billion parameters, so latency sensitive tasks such as transcription and speech synthesis fit best.

Summary

Chapter map

The whole chapter on one page. Each branch is a section, and each leaf is the idea to remember from it.

Fig. M · Chapter 3 at a glance · synthesis

Retention

Recall check

Answer in your head first, then select the card. Coming back to these in a week works better than rereading the chapter.