LLM training

The workload

Training a language model means running the same forward and backward pass over trillions of tokens, across every GPU you own, for weeks. The model's weights, gradients and optimizer state are sharded across GPUs and nodes, and every step ends with a collective operation that moves gigabytes between them. The job is only as fast as the slowest link in that chain. A training cluster is therefore a system in which the GPUs, the links between them, the links between nodes and the storage feeding them are balanced, and where any one of them being undersized wastes the rest.

The scale ranges from a few billion parameters, which fits on one node, to frontier models that need hundreds of nodes. Most customers who come to us for training hardware are between those points: a model of 7 to 70 billion parameters, trained or continued on their own data, on 8 to 64 nodes.

What it needs from the hardware

GPU memory and bandwidth first. HBM3e at 288 GB per GPU on HGX B300 lets you hold larger shards with less recomputation, and FP4 and FP8 support with the second generation Transformer Engine raises useful throughput per watt. Inside the node, NVLink and NVSwitch give every GPU full bandwidth to every other, which is what makes tensor parallelism cheap. Between nodes, the fabric needs one rail per GPU, 800 Gb/s InfiniBand or RoCE on ConnectX-8, non blocking, so that pipeline and data parallel traffic never queues behind itself.

Storage must feed the GPUs without stalls: local NVMe for checkpoints and the active dataset, and shared storage with enough throughput to restart a job from checkpoint in minutes rather than hours. Power and cooling are not an afterthought; an air cooled HGX B300 node is planned at around 14 kW, and the cluster runs at that cap for the whole job.

Recommended systems

Workstation or small server

For model development, tokenizer work, small ablations and anything under a few billion parameters, a 4 GPU AI workstation with RTX PRO 6000 Blackwell cards, 96 GB each, is enough to run real training code and find the bugs before they cost cluster time. It also serves as the dev box for a larger cluster.

PCIe GPU server

An 8 GPU PCIe server with RTX PRO 6000 Blackwell Server Edition gives you 768 GB of GPU memory in one node for a fraction of the power of an HGX system. Without NVLink, cross GPU traffic goes over PCIe, so it suits data parallel training of models that fit in one GPU, and fine tuning, better than tensor parallel training of large models.

HGX node or cluster

The 8 GPU HGX B300 AI server is the unit of serious training. One node trains models up to tens of billions of parameters; the 8 node HGX B300 AI pod and the 64 node AI cluster add an 800 Gb/s fabric, fabric switches and the acceptance tests that prove the whole thing runs at line rate. Liquid cooled HGX B300 nodes and GB300 NVL72 racks are the next step up where the facility supports them.

Sizing notes

Plan the fabric as one rail per GPU and do not share it with storage traffic. Give each node at least 128 CPU threads, 240 GB of system memory per GPU for data loading and checkpoint staging, and local NVMe of at least four times the total GPU memory. Shared storage of 1 TB usable per GPU is a sensible floor for datasets and checkpoints. Reserve one spare node per 16 for failures, because a job that waits a week for a replacement part costs more than the spare. Our sizing guide has the full set of rules.

Next step

Send the model size, the token budget, the parallelism you intend to use and the date you need to start training, and we will size the cluster, state the planning power and quote it within one business day. If you would rather spread the cost, ask for a lease quotation from one of our leasing partners alongside the purchase quotation.