A hands-on guide to local inference with llama.cpp across three GPU generations — from the 8 GB RTX 3050 to the 32 GB GDDR7 RTX 5090. What the silicon specs actually mean for tokens per second, which models fit in VRAM, and how multi-GPU builds scale.
Every request to a local LLM has two phases, and they stress the GPU in completely different ways. Understanding this is the key to reading every spec sheet on this site. Full guide →
Your whole prompt is processed in parallel to produce the first token.
This is a massive matrix-multiply workload: it is compute-bound.
What matters: CUDA core count, clock speed, and FP16/INT8 tensor performance.
Measured as t/s prompt-processing throughput.
After the first token, the model generates one token at a time, re-reading the model weights and the growing KV cache from VRAM for every single token. This is memory-bandwidth-bound. What matters almost exclusively: VRAM bandwidth (GB/s) and VRAM capacity.
llama.cpp loads GGUF models. Quantization is how you squeeze big models into consumer VRAM. The weights plus the KV cache (which grows with context length and batch size) must all fit. Full guide →
| Model size | Q8_0 ≈ | Q6_K ≈ | Q4_K_M ≈ | Q3_K_M ≈ |
|---|---|---|---|---|
| 8B | ~8.5 GB | ~6.3 GB | ~4.9 GB | ~3.9 GB |
| 13–14B | ~14 GB | ~10.6 GB | ~8.5 GB | ~6.7 GB |
| 27–32B (Qwen class) | ~33 GB | ~25 GB | ~19 GB | ~14.5 GB |
| 70B | ~70 GB | ~53 GB | ~41 GB | ~32 GB |
Rule of thumb: add ~0.5–1 GB for the KV cache of an 8K–32K context.
Q4_K_M is the community sweet spot — near-full quality at ~half the size.
You can also offload part of the model to CPU RAM (-ngl < total layers), but CPU-offloaded
layers are 10–30× slower, so partial offload only helps when there's no other option.
Where the LLM era started. All ten cards have full per-card guides, and the RTX 3090's NVLink remains the best consumer multi-GPU interconnect ever shipped. Series page →
Perf-per-watt leaps, but the VRAM story is mixed: NVIDIA moved big capacity down-market (16 GB on the 4060 Ti and 4070 Ti Super) while the RTX 4090 dropped the 3090's NVLink. Per-card guides below. Series page →
GDDR7 changes the bandwidth math: the RTX 5090 pushes 1792 GB/s — nearly double the 3090 Ti — and 32 GB of VRAM is a first for consumer cards. NVLink is gone; all multi-GPU scaling happens over PCIe 5.0. Per-card guides below. Series page →
The three rules that hold for every card on this site:
| VRAM tier | RTX 30 Series | RTX 40 Series | RTX 50 Series | Runs (Q4_K_M + KV) |
|---|---|---|---|---|
| 8 GB | 3050, 3060 8G, 3070, 3070 Ti | 4050, 4060, 4060 Ti 8G | 5050, 5060 | 8B Q8, 13B Q3–Q4 tight |
| 10–12 GB | 3080 10G, 3080 12G, 3060 12G, 3080 Ti | 4070, 4070 S, 4070 Ti | 5070, 5070 S | 13B Q4 comfortable |
| 16 GB | — | 4060 Ti 16G, 4070 Ti S, 4080 | 5060 Ti 16G, 5070 Ti, 5080 | 13B Q8, 30B MoE, 27B Q3–Q4 |
| 24 GB | 3090, 3090 Ti | 4090 | — | 27B Q4–Q5, 30B, MoE ~30B |
| 32 GB | — | — | 5090 | 27B Q8, 30B Q5+, 70B Q2–IQ3 |
Deeper dives: all cards compared → · RTX 30 Series comparison →
Series-agnostic reference pages that apply to every card on this site. Browse all guides →
llama-server -m model-q4_k_m.gguf -ngl 99 -c 8192 --host 127.0.0.1 --port 8080-ngl 99 → offload all layers to GPU-c 8192 → context length (drives KV cache size)-sm layer → split mode for multi-GPU-ngl < total → partial offload when the model doesn't fit--tensor-split 24,0 → weight per-GPU in a mixed build