Chapter 7 · Philip Kiely, Inference Engineering

Production

A fast replica is not a product. With enough traffic any single instance is overwhelmed, and that is an infrastructure problem, not a CUDA problem. This chapter covers the pieces around the model: containers, autoscaling, capacity across clouds, safe deployment and the client.

1. 7.1Package a pinned, minimal known good build in a container.
2. 7.2Scale replicas on traffic and utilization, and make cold starts fast.
3. 7.3Pool capacity across clouds for capacity, redundancy, latency and compliance.
4. 7.4Test, roll out with canaries, estimate real cost and watch the right signals.
5. 7.5Client code adds latency too: reuse sessions, go async, stream over the right protocol.

Section 7.1

Containerization

A container packages an application with its dependencies so it runs the same everywhere. Containers share the host's Linux kernel, which keeps them light. The vocabulary: a container is the running environment, an image is the executable package, a Dockerfile is the recipe and a registry stores and distributes images. Docker Hub is to images what Hugging Face is to weights.

vLLM and SGLang publish official base images, and starting from one is usually wiser than building from scratch.

7.1.1 Dependency management

7.1.2 NIMs

NVIDIA Inference Microservices are prebuilt containers for popular open models. A multi-LLM NIM runs a family of models on a supported GPU architecture, while an LLM-specific NIM is tuned for one model on one GPU setup. Use one as a starting point, a reference or an out-of-the-box service, but for maximum control build from a less opinionated base image.

Section 7.2

Autoscaling

The aim is enough resources to meet latency SLAs without paying for idle GPUs. Kubernetes runs replicas of the model container, each on its own instance. A control plane makes routing and scaling decisions and a worker plane runs the containers. Autoscaling adjusts the replica count, using utilization (memory or compute, a lagging signal) or traffic (requests in flight, which can be proactive). The two can disagree, because a few huge uncached prompts can use more GPU than many small cached ones, so use both.

Five settings define a traffic based system: minimum replicas, maximum replicas, the autoscaling window, the scale down delay, and the concurrency target. A longer scale down delay protects against spiky traffic but wastes money after a real drop. Try them.

7.2.1 Concurrency and batch sizing

Know how much concurrent traffic an instance handles. Static batching waits for a full batch, dynamic batching waits for full or a timeout, and continuous batching swaps requests in as slots free up. Engines use token level continuous batching, which minimizes latency. Larger batches raise throughput but worsen each user's latency, and the replica batch size should match the autoscaler's concurrency target. When every replica is full, add replicas, and when many run half-empty batches, scale down.

7.2.2 Cold starts

A cold start is the time to bring up a new replica, and slow ones force over-provisioning. It has four parts: GPU procurement, image loading, model loading and engine startup. Smaller images and quantized weights load faster. Weights now dwarf images, so load them separately, from storage close to the GPU rather than a third party or distant bucket. TensorRT-LLM and compiled PyTorch spend minutes compiling, so cache built engines, which only run on the same GPU type, CUDA version and dependencies.

7.2.3 Routing, load balancing and queueing

7.2.4 Scale to zero

The system can drop to zero replicas and come back on demand. That needs fast cold starts and robust queueing. It suits development, periodic agents and daily batch jobs. If you lean on it to keep a latency sensitive app cheap under light, unscheduled traffic, the product is probably not ready for dedicated infrastructure and should use pay-per-token APIs.

7.2.5 Independent component scaling

Compound pipelines have steps with different hardware needs. A voice activity model needs a far smaller GPU than the transcription model, and the LLM after it may need a node. Scale each step on its own, but keep every model in the same cluster, because cross cluster hops add up.

Section 7.3

Multi-cloud capacity management

One cluster works up to a point. A global user base needs thousands of GPUs worldwide. Siloed pools in different clouds are easy to build but cannot be used fluidly. True multi-cloud needs a global bin packing layer that treats pools as interchangeable, with self-healing and global scheduling. It unlocks capacity, redundancy, latency and compliance. A control plane handles deployment and global scaling, and workload planes serve traffic and scale locally, so one plane failing leaves others unaffected.

7.3.1 GPU procurement

Providers are hyperscalers (AWS, GCP), neoclouds (CoreWeave, Nebius) and resellers. Capacity is hard to get, especially the newest GPUs and big clusters, because the best supply goes to big customers on long reservations. Large scale inference blends sources: a reserved baseline, with on-demand and spot for peaks, spread across clusters near users.

7.3.2 Geo-aware load balancing

A global balancer avoids queues while spare capacity exists elsewhere but does not send Singapore traffic to San Francisco. A rule of thumb is five milliseconds per time zone, so New York to San Francisco is about fifteen milliseconds one way.

7.3.3 Reliability

GPUs fail often. The Llama 3 run on 16,000 GPUs for 54 days had 419 unexpected interruptions, mostly hardware, about one per 50,000 GPU-hours. One node of eight GPUs running a year is over 70,000 GPU-hours. Failures are node level: when one GPU fails others often follow, so note failures, cordon nodes and cycle pods. Cloud maintenance and outages matter too. Multi-cloud adds two postures: active-active, with several planes serving live traffic, and active-passive, with a hot standby.

7.3.4 Security and compliance

Multi-cluster infrastructure also supports data residency: a cluster near Toronto and another near New York keep Canadian data in Canada and American data in the United States, with little latency cost.

Section 7.4

Testing and deployment

Besides replica level benchmarks, test the whole system. Manual tests send synthetic traffic, load tests send volume to check scaling, and shadow traffic copies live requests to a test deployment. Testing costs engineering time and GPUs, so for example start with a random sample of shadow traffic, then a short load test. Remember that usage follows daily and weekly cycles.

7.4.1 Zero-downtime deployment

Blue-green deployment keeps two full environments and cuts all traffic over, but at scale that needs double the GPUs: 100 in blue means 100 more in green. A canary sends a small share of traffic to the new version, watches it, reverts on problems, and ramps up gradually. With autoscaling the old version scales down as traffic leaves, so cost barely rises. Make sure the new deployment has enough replicas during the ramp, or users see queued latency spikes.

7.4.2 Cost estimation

Public API cost is linear: price per million tokens times tokens. Dedicated cost depends on batch size, traffic saturation and sequence lengths. It is easier to convert your token spend into a total and compare that with dedicated costs than to back out a per-token price. Use a horizon of at least a week, and add engineering time to get total cost of ownership.

7.4.3 Observability

Monitor inference like any mission-critical service, and read the metrics together, because a latency spike could come from volume or from longer inputs. Add server and audit logs, and integrate with Grafana, Datadog, PagerDuty and Sentry instead of building a silo.

Section 7.5

Client code

The other side of every call is easy to overlook. The industry standard is the OpenAI SDK, and libraries such as LangChain, Vercel AI SDK, LiteLLM and LlamaIndex also act as clients. Any of them can add latency, and real-time apps may need protocols beyond HTTP.

7.5.1 Client latency overhead

Setting up a session takes a few dozen milliseconds. With a 300 ms P95 target, a TLS handshake alone can eat at least ten percent of the budget before inference starts. Reuse sessions, which the OpenAI SDK does quietly, and do the same when you write a client for a nonstandard modality.

7.5.2 Asynchronous inference

Throughput oriented work like bulk documents or corpus embedding should be fire and forget. Synchronous calls time out after a few minutes. Async jobs are acknowledged immediately and deliver the result to a webhook, with limits measured in hours, and strong queues make this robust.

7.5.3 Streaming and protocols

Streaming makes applications feel instant. HTTP streaming is enough for text, but live voice and video need bi-directional streams. WebSockets suit unstructured real-time data like audio, and a server allows a configured number of connections before new ones wait for a slot or a new replica. gRPC streams structured data against a predefined schema, which removes parsing but adds a little validation overhead.

Summary

Chapter map

The whole chapter on one page. Each branch is a section, and each leaf is the idea to remember from it.

Retention

Recall check

Answer in your head first, then select the card. Coming back to these in a week works better than rereading the chapter.