Chapter 6 · Philip Kiely, Inference Engineering

Modalities

Models can take and produce text, images, audio, video and vectors, yet nearly all of them fall into two archetypes: autoregressive token generation or iterative denoising. The LLM toolbox carries over to most, but each modality changes what you measure and what you optimize.

1. 6.1VLMs add image tokens. Prefill and the KV cache grow, so every LLM technique helps.
2. 6.2Embeddings split into backfill throughput and low-latency lookup. Scale out with one replica per GPU.
3. 6.3ASR is an encoder-decoder. Streaming and parallel chunking drive the gains.
4. 6.4TTS is a fine-tuned LLM plus an audio decoder. Measure time to first byte and concurrent streams.
5. 6.5Image models are compute bound, so choose kernels, fuse, compile and trim guidance.
6. 6.6Video is the heaviest. Attention dominates, and context parallelism spreads it over eight GPUs.

Before 6.1

Modalities and how to measure them

A modality is the pair of input and output types a model handles: text and images to text for vision language, audio to text for transcription, text to audio for speech, text to image or video for generation, and so on. Many of the same engines and techniques apply, but each modality needs its own idea of latency. A single audio token is meaningless, so instead of time to first token you measure time to first word or sentence.

Section 6.1

Vision language models

A VLM takes images or video plus a prompt and returns text. It has a large LLM and a small vision encoder, for example a 2B encoder beside a 673B LLM in Mistral Large 3. The encoder is small but critical, and its architecture varies, so runtime support is more fragmented, which raises the importance of vLLM and SGLang. A high-resolution image adds roughly a thousand visual tokens, so the main challenge is a longer input and a bigger KV cache.

VLMs also add a quality and speed trade: downsampling. A high-resolution image costs about four times the tokens of a low-resolution one. Single images rarely need it, while multiple images or clips may.

6.1.1 Video input

Video has motion and sometimes audio, which most VLMs cannot hear, so audio is transcribed separately. Quality needs the whole clip in one call. At 24 frames per second and about 1,000 tokens per frame, four seconds approaches 100,000 tokens, so downsampling resolution and frame rate is practically obligatory, and clips stay short.

6.1.2 Omni-modal models

Omni models accept and produce many modalities, but smaller specialists are often faster and more accurate. Built-in text recognition lags dedicated OCR models that are a fraction of the size. Production VLM work often chains preprocessors for PDFs, OCR and audio, and each stage should be optimized and scaled on its own.

Section 6.2

Embedding models

An embedding model maps variable length input to a fixed length vector that captures meaning, so closeness can be computed with simple math. They power agent memory, RAG, search and recommendations. Two traffic profiles exist, and you should decide which you serve, or build a separate system for each if traffic justifies it.

6.2.1 Architecture

Models are either BERT style, encoder only and usually under 1B parameters, good for simple latency sensitive tasks like classification, or LLM based, up to about 8B and much more capable. Vector length trades speed and quality. Matryoshka representations keep information in shorter vectors. Dimensionality barely changes inference time but changes storage, retrieval and similarity cost. Vectors from different models cannot be compared, even at equal length.

6.2.2 Inference

Runtimes include vLLM, SGLang, Infinity and Hugging Face TEI, but the best performance comes from adapting TensorRT-LLM, with its XQA attention kernel and fusion. FP8 weights give gains with minimal quality loss. Check quality by running the same inputs through both models and comparing output vectors with cosine similarity, looking for at least 99 percent. Prefix caching and disaggregation do not apply because tokens are processed in parallel, and multi-GPU parallelism is not worth it for small models. Scale horizontally with one replica per GPU. Batches can hold dozens or hundreds of inputs, and a robust queue is needed for backfills and surges.

Section 6.3

ASR models

ASR turns audio into text. The popular open model is Whisper, an encoder-decoder with a 1.55B parameter largest size, so it runs well on MIG slices of an H100. The encoder reads a log-Mel spectrogram, and the decoder, an LLM-like autoregressive transformer, takes most of the time. TensorRT-LLM gives in-flight batching and fast kernels for it.

6.3.1 Real-time chunks

For live use, aim for about 200 ms per chunk, roughly human reaction time. The runtime is already tuned, so gains come from streaming at the API layer: a WebSocket carries audio in and text out, and a voice activity detection model segments the stream into chunks. Processing chunks on one GPU lets each output seed the next.

6.3.2 Long files

Whisper handles 30 second chunks, so long files need a pipeline. Performance is the real-time factor: an hour transcribed in 30 minutes is 2x, and an optimized system can do an hour in under four seconds, 1000x. Run VAD on its own hardware to cut silence and split at speech boundaries, transcribe chunks in parallel, and stitch by timestamp. RTF scales roughly linearly with GPUs.

Parallel chunks lose the previous-output prefix, but you can detect hallucinations by compression ratio and words per minute, and then re-run the chunk at a higher temperature to break repetition, or re-chunk into smaller pieces.

6.3.3 Diarization

Diarization labels who spoke when. It is a pipeline of classic ML models for segmentation, embedding and clustering, such as pyannote audio. Optimize each model and the orchestration with PyTorch and compilation. Even a tuned system takes at least twice as long as transcription.

Section 6.4

TTS models

Modern TTS models are fine-tuned LLMs. Orpheus TTS comes from Llama 3.2 3B, with its vocabulary extended by tens of thousands of audio tokens, so LLM tooling applies. Weights and KV cache can go to FP8, and MIGs on H100s work well. An audio decoder, compiled in PyTorch and using dynamic batching with a short timeout such as 15 ms, converts audio tokens to a waveform. In-flight batching is not possible there, and it can become a bottleneck.

The key metrics are TTFB, the equivalent of TTFT (about 150 ms is reachable on one H100 for Orpheus), the more user oriented time to first sentence, and TPS. Real-time speech needs only about 80 to 100 tokens per second per stream, and more speed adds nothing. Instead, raise the number of concurrent real-time streams to lower cost per user.

6.4.1 Streaming TTS

Again the biggest win is infrastructure: stream over WebSockets and match batch size to the number of active sockets after load testing how many streams the engine sustains. TTS is rarely batch, and quality degrades on inputs past about 30 seconds.

6.4.2 Speech-to-speech

Most voice systems cascade ASR, an LLM and TTS, with VAD and embedding helpers. Speech-to-speech models such as gpt-realtime merge these into one model because the three share similar decoder architectures. At publication no open option is commercially viable, and closed ones are less capable and costlier than cascades, but research is active.

Section 6.5

Image generation models

Image models differ from LLMs in architecture (pipelines of small models in latent space, mostly denoisers), tooling (SGLang Diffusion and vLLM Omni are brand new, so work often sits in PyTorch or TensorRT), constraints (10 to 20 times smaller than frontier LLMs and compute bound) and quality, which is hard to score programmatically and is usually judged by humans choosing between images.

6.5.1 Kernels

Model cards use diffusers with few optimizations. Even though the work is compute bound, you often need memory efficient kernels and fusion to reach that limit.

Attention is the key kernel, with FlashAttention 3 on Hopper and 4 on Blackwell beating version 2. Small kernels like RMSNorm are good fusion targets, and GEMM kernels are generally safe to quantize to FP8 for twice the Tensor Core FLOPS. Torch compilation fuses automatically, takes minutes so cache the result, and must be done on the same GPU type you deploy on.

6.5.2 One weird trick

Time scales linearly with steps. Each step runs the denoiser twice, with and without guidance. Early steps set the outline and later steps fill details, so turn guidance off partway: skipping it for the last 20 of 50 steps leaves 80 passes instead of 100, usually with quality intact.

Section 6.6

Video generation models

Video is the most demanding modality. Run it on Blackwell or Rubin when possible, for memory capacity to support context parallelism, fast Tensor Cores and microscaling formats. It is compute bound denoising over a 3D latent with about 50 steps, with far more data per step. Batching does not help, so models run on a full eight GPU node at batch size one, and the only way to improve throughput and cost is a faster model. Attention is 70 to 80 percent of compute time.

6.6.1 Attention optimization and quantization

Start by testing kernels from FlashAttention, DeepGEMM, CuTe and CUTLASS. Caching reuses earlier computation, and can make generation 30 to 40 percent faster, but quality varies from negligible loss to unusable, so test. Quantization focuses on attention because for video the Tensor Core FLOPS matter, not memory traffic.

6.6.2 Context parallelism

Eight GPUs usually run context parallelism rather than tensor parallelism. Every GPU holds the full weights, which is feasible on a B200, and the attention computation is split. In ring attention, each GPU holds a piece of the context and passes intermediate results around a ring. Attention heads are independent, so they can run separately and combine afterward. Other steps parallelize too: VAE latent decoding takes three to five percent of the time and can run across GPUs.

Summary

Chapter map

The whole chapter on one page. Each branch is a section, and each leaf is the idea to remember from it.

Retention

Recall check

Answer in your head first, then select the card. Coming back to these in a week works better than rereading the chapter.