The new single-card king. 32 GB of GDDR7 on a 512-bit bus at 1792 GB/s — nearly double the 3090 Ti's bandwidth, and the first consumer card where 27B runs at Q8.
Token generation is bandwidth-bound, so this number is the card: ~1.9× the 3090 Ti, ~2× the 3090. The results: ~200–250 t/s on 8B, ~105–135 t/s on 13B Q4, ~35–48 t/s on 27B Q4 — a model class that ran 18–25 t/s on the 3090 now runs at 3090-MoE speed. Prompt processing (~1600–2200 t/s on 13B) is 21760-core scale: long documents digest in well under a second.
-ot output=CPU resolves — the first consumer card where this is a discussion;No other consumer card — 30, 40 or 50 series — comes close. The 3090's 24 GB was the ceiling for three years; this is +33% of it.
The 5090 dropped NVLink entirely. Multi-GPU still works via tensor split, but the split tax grows from ~10–15% (3090 NVLink) to ~25–35% (PCIe 5.0). 2× 5090 (64 GB) is still the new money path to 70B Q4 with room — just budget for the tax and for 1.15 kW of GPU power.
Before the numbers: if your model ships a built-in multi-token-prediction head
(e.g. Qwen 3.5/3.6/3.8, Gemma 4), generation can be made faster with
--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1 — on this card that's worth
+60–140% (measured) on token generation; community A/Bs on this card measured +119–142% at 262K context (n-max 2); gains shrink at short context — treat 8K numbers as the lower end of the range. It costs ~0.6–2 GB extra VRAM and
does not change output quality, and it only applies to MTP-capable models — Llama-class
models are unaffected. Details and the tuning rules: MTP guide →
| Model | Size on GPU | Token gen | Token gen (MTP est.) | Prompt proc. | Fits? |
|---|---|---|---|---|---|
| 8B Q8_0 | ~9.7 GB | ~200–250 t/s | ~320–500 t/s | ~1600–2200 t/s | ✅ 64K ctx |
| 13B Q4_K_M | ~8.5 GB | ~105–135 t/s | ~170–270 t/s | ~1800–2400 t/s | ✅ 64K+ ctx |
| 27B Q4_K_M | ~19 GB | ~35–48 t/s | ~56–95 t/s | ~900–1200 t/s | ✅ 16K+ ctx |
| 27B Q5_K_M | ~22 GB | ~30–40 t/s | ~48–80 t/s | ~800–1100 t/s | ✅ 8–16K ctx |
| 27B Q6_K | ~25 GB | ~25–32 t/s | ~40–64 t/s | ~700–1000 t/s | ✅ 8K ctx |
| MoE 30B-A3B (Q4) | ~18 GB | ~140–190 t/s | ~220–350 t/s | ~1500–2000 t/s | ✅ fast (3B active) |
| 70B IQ2_XXS | ~24 GB | ~10–13 t/s | ~16–25 t/s | ~500–700 t/s | ⚠️ 4–8K ctx, low quality |
MTP est. = rough prior with --spec-type draft-mtp for models that ship MTP heads (Qwen 3.5/3.6/3.8, Gemma 4); Llama-class models get no MTP speedup. Actual gains depend on model, quant, context length, llama.cpp build and your card — sweep --spec-draft-n-max, don't trust the column blindly. MTP guide →
Estimates for full GPU offload (-ngl 99), ~8K context, batch 2048. Token generation scales roughly with memory bandwidth; individual runs vary by model architecture (GQA vs MHA, MoE) and llama.cpp build. MoE token gen depends on active params, not total — treat as a bonus speed tier.
| Quant | GGUF size | 1× 5090 (32 GB) | 2× 5090 (64 GB) |
|---|---|---|---|
| Q4_K_M | ~19 GB | ✅ ~35–48 t/s, 16K+ ctx | ✅ (overkill) |
| Q5_K_M | ~22 GB | ✅ ~30–40 t/s, 8–16K ctx | ✅ ~50–62 t/s, 16K+ ctx |
| Q6_K | ~25 GB | ✅ ~25–32 t/s, 8K ctx | ✅ ~44–55 t/s, 16K+ ctx |
| Q8_0 | ~33 GB | ⚠️ ~33 GB weights — -ot output=CPU, 4–8K ctx | ✅ ~38–48 t/s, 8–16K ctx |
The 5090 is the only single card on this site where 27B runs at Q6, and the first where Q8 is a conversation instead of a hard no — the 33 GB weight file misses 32 GB by a sliver, which parking the output embedding on CPU (-ot output=CPU) closes at short context. The 2× build (64 GB) removes every restriction: Q8 at 8–16K, and 70B Q4_K_M at ~30–38 t/s — the new-money equivalent of a 3090 pair running 70B.
-c 16384 fits; Q6 wants -c 8192 plus KV quantization (--cache-type-k q8_0 --cache-type-v q8_0).-ot output=CPU -c 8192 — the output embedding on CPU reclaims the GB it needs.No NVLink — the split runs over PCIe 5.0 with a ~25–35% tax (a 3090 pair's NVLink costs ~10–15%). 32 GB per tile: 2× reaches 64 GB, the 70B Q4 line.
| Build | GPU power | VRAM | 27B Q8_0 | 70B Q4_K_M | 70B Q5–Q6 |
|---|---|---|---|---|---|
| 2× 5090 | 1150 W | 64 GB | ~45–55 t/s, 16K+ ctx | ~30–38 t/s, 8–16K ctx | ~24–30 t/s, 8K ctx |
| 3× 5090 | 1725 W | 96 GB | (overkill) | ~38–46 t/s, 16K+ ctx | ~28–36 t/s, 32K+ ctx |
2× (64 GB) is the build that replaces 2× 3090 for 70B: more VRAM headroom, much faster per token, at the cost of the PCIe tax. 3× (96 GB) is the long-context flagship — 70B Q5 at 32K+ is a thing on this rig. Combined bandwidth (3584 GB/s) makes it the fastest 70B rig below server land.
When a model doesn't fit, -ngl N keeps the last layers on the CPU instead.
The rule to internalize: offloaded layers run at CPU speed, not a percentage of GPU
speed. Practical patterns: drop just the last 2–4 layers to the CPU to reclaim ~0.5–1 GB
of VRAM (nearly free); keep MoE experts on CPU with --n-cpu-moe; park the output
embedding with -ot output=CPU to save another ~0.5–1 GB on big models. Budget 32 GB
of system RAM for 27B-class offload, 64 GB for 70B. Never offload the KV cache — the conversation
gets unusably slow.
llama-server -m 70b-q4_k_m.gguf -ngl 99 -sm layer -c 16384 --tensor-split 1,1llama-server -m 70b-q5_k_m.gguf -ngl 99 -sm layer -c 32768 --tensor-split 1,1,1
llama-server -m qwen3-27b-q5_k_m.gguf -ngl 99 -c 16384 -b 2048 -ub 2048 --host 127.0.0.1 --port 8080
The single-card LLM card of this era, full stop: 1792 GB/s runs 27B at 3090-MoE speed, 32 GB opens the Q6/Q8 line, and MoE 30B is effectively free at ~150 t/s. The price is 575 W, the absence of NVLink, and launch-window money — but there is no second option that comes close, in any generation. For 27B single-card work it's the default; for 70B, the 2× build is the new flagship rig — and the 3090 pair remains its used-market shadow at less than half the cost.