How to Size RAM and CPU for Hosting 7B to 70B Parameter Models
The number-one self-hosting failure is not the software. It is ordering a server, downloading the model, and finding out the box cannot hold it, or holds it only by swapping, which turns inference into a slideshow. Sizing for LLM hosting is arithmetic you can finish before you buy anything. This article does that arithmetic for the 7B to 70B class, on GPU and on CPU.
Start with the weights, not the vibes
llama.cpp's own documentation publishes the anchor numbers for Llama 3.1: 8B goes from 32.1 GB in original format to 4.9 GB at Q4_K_M, and 70B goes from 280.9 GB to 43.1 GB. That is roughly 4.85 bits per parameter at Q4_K_M, about one byte per parameter at Q8_0, and two bytes at FP16. The heuristic falls out of the math: a model's Q4_K_M file weighs about 0.6 GB per 10 billion parameters, a Q8 file about 1 GB per 10 billion.
So a 7B or 8B at Q4_K_M is a 5 GB file. A 32B-class model lands near 20 GB. A 70B at Q4_K_M is roughly 43 GB, and at Q5_K_M community publishes put it near 50 GB. Write these numbers down before touching a shopping cart, because everything else in the budget is an add-on. MoE models bend the rule, because only active parameters move per token, but dense 7B to 70B hosting, the bread and butter of rented servers, follows the arithmetic exactly as written here.
The context tax: KV cache
Weights are the fixed cost. The KV cache scales with the conversation. Every generated token stores key and value vectors for every layer, and published calculators put the 70B class at a few gigabytes for 8K context while a 128K context can demand 10 to 20 GB on top of the weights. That is why identical model files run fine in short chat and crash on a RAG prompt with a 100-page context.
Budget in three lines: model file, plus expected KV cache at your real context length, plus 1 to 2 GB of runtime overhead. Multiply the cache portion by your concurrency if you serve several sessions from one process. If the total lands within about 90 percent of your available memory, you sized it. If it does not, shorten context or step down the quant, in that order.
CPU hosting changes the question to bandwidth
Running on system RAM instead of VRAM is legitimate, and it is constrained by a different resource. CPU inference is memory-bandwidth bound: the engine reads essentially every weight once per generated token. A 7B Q4 model reads around 4 to 5 GB per token, so at tens of gigabytes per second of practical DDR bandwidth you get single-digit tokens per second. Cores beyond your memory channels' ability to feed them add nothing, and llama.cpp's maintainers specifically recommend fewer threads than SMT suggests.
For CPU-only serving, RAM capacity gets you the model loaded, but bandwidth determines whether it is usable. A dual-socket server or a platform with more active memory channels runs the same model faster at the same clock. Budget capacity first, then buy the widest memory configuration that capacity allows. Disk enters the picture as a fourth constraint: your model files live there, they are tens of gigabytes each, and cold-start load time is a storage-bandwidth number, not a memory one.
Worked examples and the ECC footnote
Example one: 7B Q4_K_M on CPU. About 5 GB weights, a couple GB of cache at 8K, so a 16 GB instance is comfortable and 8 GB is tight but feasible. Example two: 70B Q4_K_M fully in VRAM needs roughly 45 to 50 GB at 8K context, which means 48 GB of accelerator or two 32 GB cards, and layer-splitting lets llama.cpp spill the overflow into system RAM at a bandwidth penalty. Full-GPU sizing for that model wants 64 GB of host RAM with headroom for OS and cache.
One footnote matters for rented boxes: ECC. DDR5 chips correct single errors on-die without telling the controller, and server RDIMMs add system-level ECC with device correction designed to survive a whole failing chip. A silent corruption in a 43 GB model file does not crash, it just makes your LLM confidently weird. For always-on hosting, ECC RAM is a spec worth paying for, not enterprise theater.
Sizing is just accounting: the quant sets the weight bill, the context sets the cache bill, concurrency multiplies both, and bandwidth decides whether CPU hosting is chat-viable or batch-only. Do the arithmetic once per model class and you can rent the right box without a single swap-file surprise. Predictable compute starts with specs you chose on purpose.
Gear we recommend for AI workloads
Crucial Pro 128GB Kit (2x64GB) DDR5 RAM, 5600MHz (or 5200MHz or 4800MHz) Desktop Gaming Memory UDIMM, Compatib
Top-rated on Amazon
View on Amazon
Lenovo ThinkPad T16 Gen 4 AI Business Laptop 16” FHD+ Display, Intel Ultra 7 255U (Beats i7-1355U), 64GB DDR5
Top-rated on Amazon
View on Amazon
GMKtec EVO-X2 AI Mini PC AMD Ryzen Al Max+ 395 Up to 5.1GHz, 16C/32T | Mini Gaming Computers 64GB LPDDR5X 8000
Top-rated on Amazon
View on AmazonAs an Amazon Associate we earn from qualifying purchases.
