Fine tuning

The workload

Fine tuning takes a model somebody else trained and teaches it your domain, your tone, your tools or your task. It is the most common way an organization gets a model that is actually useful, and it is a far smaller job than pretraining because the hard part, learning language, is already done. The work comes in three sizes. Parameter efficient methods such as LoRA and QLoRA train a small set of adapter weights and leave the base model frozen; they run on surprisingly modest hardware. Full parameter fine tuning updates every weight and needs memory for the optimizer state as well, which is several times the model size. Continued pretraining on a large domain corpus sits between fine tuning and training proper and is sized like a small training job.

Fine tuning runs are short, hours to days, and they are repeated often as data and evaluations improve. That favors hardware that is available on demand inside your own organization rather than queued.

What it needs from the hardware

Memory is the gating resource. A LoRA run on a 70 billion parameter model in 4 bit needs around 40 to 50 GB of GPU memory and fits one high memory card. Full fine tuning of the same model in mixed precision needs weights, gradients and optimizer state, which is well over a terabyte, and so needs an 8 GPU HGX node or sharded training across several PCIe GPUs with the optimizer state offloaded. Between those sit the 7 to 13 billion parameter models that are fully fine tuned on two to four GPUs.

Interconnect matters when a run is sharded across GPUs. PCIe is fine for LoRA and for data parallel runs of models that fit one GPU; NVLink pays for itself once the model and optimizer must be split. Storage needs are moderate: a fast NVMe for datasets and checkpoints, and shared storage if several people run jobs on the same box.

Recommended systems

Workstation or small server

A 4 GPU AI workstation with RTX PRO 6000 Blackwell cards, 96 GB each, runs LoRA and QLoRA on models up to 70 billion parameters and full fine tuning on models up to about 13 billion, under a desk or in a lab. The liquid cooled 4 GPU workstation runs the same work at full power without the fan noise.

PCIe GPU server

For a team, a 4 GPU 4U PCIe server or an 8 GPU PCIe server with RTX PRO 6000 Blackwell Server Edition gives several people their own GPUs for parallel experiments, or one person eight GPUs for a sharded run, in a rack rather than an office. Eight 96 GB GPUs full fine tune models in the 30 billion parameter class comfortably and larger ones with offloading.

HGX node or cluster

Full fine tuning of 70 billion parameter and larger models, or continued pretraining on tens of billions of tokens, calls for the 8 GPU HGX B300 AI server. 2.1 TB of HBM3e over NVSwitch holds model, gradients and optimizer state for a 70B model without tricks, and the node later serves the model it trained. Several nodes form a pod when the corpus is large.

Sizing notes

Decide the method first, because it sets the memory. For LoRA, plan one GPU per concurrent experiment with memory for the quantized base model plus about 30 percent. For full fine tuning, plan 16 to 20 bytes per parameter across the GPUs in the run. Give the host 240 GB of system memory per GPU if you intend to offload optimizer state, and local NVMe of four times the GPU memory for checkpoints, which fine tuning writes often. A 100 Gb/s connection to shared storage is plenty. The rules are collected in the sizing guide.

Next step

Tell us the base model, the method, how many people will run jobs at once and where the system will live, and we will quote a configuration within one business day. If the day to day work fits a PCIe server but one large fine tuning run needs an HGX node, say so and we will quote both so you can see the difference.