Hardware for Local LLMs: Sizing a GPU Server Properly

Most teams that want to run a language model on their own premises start by asking which graphics card to buy. That is the wrong first question. A parts list says little until you know which model has to serve how many people, working with documents of what length. This guide sizes the hardware from the use case, working backwards from the workload to the server. Status: October 2026.

What determines the hardware needs of a local LLM?

Three quantities decide it: the size of the model, the number of people using it at the same moment, and the context length of each request. The first two drive GPU memory, the third drives both memory and compute time. Power supply, chassis and the rest follow afterwards.

Before any purchase, write down three numbers.

• Model: Which parameter count and which precision (BF16, FP8, 4-bit) does the task need?
• Concurrent requests: Not headcount, but the number of requests running at the same instant. A company with 50 staff often has only a handful in flight at once.
• Context length: How much text goes into each request? A short chat takes a few thousand tokens. Analysing a stack of long documents takes many times that.

From these three inputs you can compute the memory budget before comparing a single card. This matters because memory is the hard limit: whatever does not fit does not run.

How much GPU memory do the model weights need?

The weights take roughly the parameter count times the bytes per parameter. A 72-billion-parameter model needs about 144 GB in BF16 (2 bytes), about 72 GB in FP8 (1 byte) and roughly 36 GB with 4-bit weights. Runtime overhead and cache come on top.

This is a back-of-the-envelope estimate, not a vendor figure. The exact parameter count, file format and inference software shift the numbers slightly. It is good enough for planning as long as you keep a buffer.

According to its quantization documentation, the inference engine vLLM supports FP8, AWQ, GPTQ and INT4 and INT8 schemes, among others. Quantisation trades precision for a smaller memory footprint. How much it matters for your task is something only a test on your own questions and documents can show.

What is the KV cache, and why does it grow with users and context?

The KV cache stores intermediate values for every token seen so far in a request, so the model does not recompute them for each new token. It grows linearly with context length and is held separately for each request. Ten parallel requests need ten caches, while all of them share one set of weights.

You can derive its size from the model configuration. For every token, keys and values are stored in every layer:

KV bytes per token = 2 × layers × KV heads × head size × bytes per value

The configuration of Qwen2.5-72B-Instruct lists 80 layers, 8 key-value heads, 64 attention heads and a hidden size of 8192, which gives a head size of 128. With 2-byte values that is 2 × 80 × 8 × 128 × 2 = 327,680 bytes, or 320 KiB per token. The smaller Qwen2.5-7B-Instruct has 28 layers, 4 key-value heads, 28 attention heads and a hidden size of 3584, again a head size of 128. That comes to 56 KiB per token.

Scaled to realistic contexts, the 72B model needs:

The table shows why long documents can dominate the plan. At 32,000 tokens per request, ten parallel users need more cache than a single card has in total memory. The figures assume a cache with 2 bytes per value (BF16).

How do I get from a use case to a GPU size?

Add up the weights, the KV cache for your planned concurrency, and a reserve for runtime and activations. The result is your minimum memory. The total should sit comfortably below the usable memory of the card or card group, not right at its edge.

A worked example: a department wants to run a 72B-parameter model in FP8, with up to 8 concurrent requests of 8,000 tokens each.

1. Weights: about 72 GB.
2. KV cache: 8 × 2.5 GiB, so about 20 GiB.
3. Reserve: a buffer for runtime, the CUDA context and activations, which you set after measuring.

Even before the reserve, the total is about 92 GB. A 48 GB card is out. A 96 GB card would be tight. Two cards, or a smaller model, would leave headroom. The same calculation for a 7B model in BF16 gives about 14 GB plus 8 × 8,000 × 56 KiB, a little over 3 GiB, which fits on almost any professional card.

The vLLM documentation describes what happens when the cache is too small: if there is not enough room for all batched requests, the engine may preempt requests and recompute them later, which makes responses slower. Among its suggested remedies are giving the cache a larger share of GPU memory and limiting the number of concurrent sequences.

Which GPU classes are worth considering?

For on-premises use, three classes matter: professional cards with 48 to 96 GB, data-centre cards with high-bandwidth memory, and groups of several cards. What decides the choice is memory size, memory bandwidth, power draw and form factor, not raw compute alone.

The following figures come from the manufacturer's product pages (retrieved October 2026):

Sources: L40S, RTX PRO 6000, H100. "Not stated" means the page does not give that figure. Before buying, check the current datasheets, availability, and the cooling and power limits of your server room.

Two things stand out. First, the H100 variants list 3.35 to 3.9 TB/s of memory bandwidth, well above the 864 GB/s of the L40S. Second, cards with 80 to 96 GB only just hold a 70B-class model in FP8, leaving little room for cache.

Why does memory bandwidth decide the speed?

To produce each token, the GPU essentially has to read all the weights from memory once. The speed of a single request is therefore capped at memory bandwidth divided by the model size in bytes. In this phase compute is usually not the bottleneck.

A rough calculation with the bandwidths above:

• 72 GB of weights (FP8) on an H100 SXM at 3.35 TB/s: an upper limit of about 46 tokens per second for one request.
• 36 GB of weights (4-bit) on an L40S at 864 GB/s: an upper limit of about 24 tokens per second.

These are theoretical ceilings. Real figures are lower, because the KV cache is read as well and software adds overhead. The counterweight is batching: when several requests are processed together, they share the cost of reading the weights. Total throughput rises sharply while each individual request becomes a little slower. How many users a server really handles is a separate question that depends on measurements with your own load.

When do I need more than one GPU?

You need several GPUs when the model and its cache do not fit on one card, or when one card cannot deliver the throughput. The usual technique is tensor parallelism, which splits the parameters of each layer across cards.

The vLLM documentation puts it this way: tensor parallelism shards model parameters across multiple GPUs within each model layer. That lowers the memory needed per card and frees space for cache, but adds synchronisation overhead. Some planning rules follow:

• Two 48 GB cards are not the same as one 96 GB card. The link between the cards (bus, possibly NVLink) affects speed.
• The number of cards has to suit both the model architecture and the inference software, so check the software's requirements before you buy.
• More cards mean more power, more heat and more points of failure.

The alternative is several independent instances, each on its own card. That is more robust, but it requires the model to fit on one card.

What else belongs in the server besides the GPU?

A GPU alone does not make a server. You also need enough system memory to load models, fast NVMe storage for model files, adequate PCIe connectivity, a power supply with headroom and cooling that can handle a sustained load.

In more detail:

• System memory and disk: Models are large files, loaded at start-up. Leave room for several model versions.
• Power and cooling: The power figures above are per card. Several cards in one chassis only run continuously with proper airflow or liquid cooling.
• Form factor: The server and workstation versions of a card differ in design and power draw (see the table). Settle the space and airflow questions early.
• Network: If several departments use the server, network connectivity and access control are part of the plan.
• Operations: Monitoring of utilisation, memory and temperature, driver and inference-software updates, and a plan for failures.

On data protection: running a model locally avoids sending data to external services, which makes it possible to operate a setup in line with the GDPR. Whether that holds depends on access rights, logging and a deletion policy, and it does not replace legal advice. The page on sovereign AI puts hardware choices in the wider context of digital sovereignty.

What are the steps to follow?

Start from the use case, calculate the memory, test on rented or borrowed hardware, and only then buy. That order prevents you from purchasing hardware that does not fit the model you end up using.

1. Pin down the use case: Which task, which text lengths, how many simultaneous requests at peak?
2. Choose a model: Start with the smallest model that solves the task in your own test. Bigger models need bigger hardware.
3. Calculate memory: Weights plus KV cache plus reserve, as shown above.
4. Decide on precision: Test BF16, FP8 or 4-bit against answer quality instead of assuming.
5. Compare candidates: Memory size, bandwidth, power draw, form factor and availability, using current datasheets.
6. Measure load: Run the chosen model under realistic load on a rented or borrowed GPU and record response times with parallel requests.
7. Set reserves: Allow for growth in users and context length. A server should not start life at its limit.
8. Plan operations: Monitoring, updates, backups of model files, and clear ownership.

What are the limits and risks?

The calculations in this article are planning aids, not guarantees. Real values depend on the model, software version, driver and load. Hardware also ages quickly, because new model architectures and quantisation methods shift the requirements.

• Quality loss from quantisation: Smaller weights save memory but can cost accuracy. Without your own test you cannot tell.
• Growing demands: Longer contexts and more users after launch will overwhelm a card that was sized too tightly.
• Lead times and availability: Datasheets say nothing about delivery. Ask your reseller.
• Operating effort: Your own GPU server needs maintenance and know-how. Whether that beats an external service is a question of cost that depends on your situation.
• Not every task needs a large model: For many well-defined jobs a small model that fits on one card is enough.

How do I get started?

Start small. Express the use case in three numbers, work through the memory budget, and test the model on a single GPU before investing in a multi-card system. That teaches you what load really arrives.

A pilot with a few users and real documents helps. Measure memory use, response time and the number of simultaneous requests. With those figures, capacity planning becomes reliable. For an overview of where agents running on local models are used, see the page on AI agents. The choice between an own server and an external API comes down to load, data-protection requirements and operating effort.