LLM inference

The workload

Inference is the business end of a language model: a request arrives, the model reads the prompt, and it generates tokens one at a time until it stops. In production the server handles hundreds of these at once, batching them together so the GPU is never idle. Two numbers matter to the people paying for it: throughput, the tokens per second the server produces across all users, and latency, the time to the first token and the time between tokens for each user. Both are set by the hardware, the model size and the serving stack, and the hardware decides the ceiling.

Inference workloads vary more than training ones. A chat assistant for 50 staff, an API serving millions of requests a day, and a batch pipeline that summarizes documents overnight all run the same model and need very different servers.

What it needs from the hardware

GPU memory first, because the model's weights must fit, and then the KV cache for every concurrent request has to fit beside them. A 70 billion parameter model in FP8 needs around 70 GB for weights before any cache; in FP4 it needs half that. Memory bandwidth decides how fast tokens come out once the batch is formed, which is why HBM3e parts lead on tokens per second per GPU and GDDR7 parts lead on tokens per second per watt and per rack unit.

Cross GPU bandwidth matters when a model is split across GPUs with tensor parallelism. NVLink on HGX systems makes that split almost free; on PCIe systems the split is possible but costs latency, so the better approach is one model per GPU or per pair. Beyond the GPUs, inference needs fast network ingress, enough CPU to run tokenization and request handling, and little storage, since models load once.

Recommended systems

Workstation or small server

A 2 GPU 2U inference server with RTX PRO 6000 Blackwell Server Edition cards carries 192 GB of GPU memory, enough to serve a 70 billion parameter model in FP8 with room for cache, from a short depth chassis that fits an office rack or an edge site. A 1 GPU 1U edge inference server covers smaller models at branch sites.

PCIe GPU server

The 8 GPU PCIe server with RTX PRO 6000 Blackwell Server Edition is the workhorse of production inference. Eight independent GPUs serve eight model replicas, or four pairs for larger models, at a planning power well under half that of an HGX node. For models that benefit from HBM bandwidth without NVLink, the 8 GPU PCIe server with H200 NVL pairs GPUs over NVLink bridges and carries 141 GB per GPU.

HGX node or cluster

When one model must serve the highest token rates, or when the model is too large for a pair of PCIe cards, the 8 GPU HGX B300 AI server splits it across eight GPUs over NVSwitch with 2.1 TB of HBM3e. Platforms serving several frontier scale models use pods of HGX nodes behind a load balancer, and the GB300 NVL72 rack for the largest models of all.

Sizing notes

Start from the model and the quantization, add the KV cache for your target concurrency and context length, and pick the GPU memory that holds it with 20 percent spare. Then decide whether throughput or latency is the goal: throughput favors more GPUs with independent replicas, latency favors fewer, faster GPUs with the model split over NVLink. Give the host 6 CPU threads and 24 GB of memory per GPU for the serving stack, and 2 x 100 Gb/s or better front end networking on a busy API node. The inference server guide walks through the arithmetic with examples.

Next step

Tell us the model, the quantization you plan to use, the expected concurrency and the latency you need, and we will recommend a server and a GPU count with the working shown. Quotations are returned within one business day. If you are comparing against cloud GPU pricing, the same sizing applies.