Choosing a GPU server for LLM inference
A practical method for sizing an inference server from the model, the quantization and the concurrency you expect, with worked examples.
In this guide
- Start with the model, not the server
- Step 1: how much GPU memory the weights need