The 3090 overclocked by NVIDIA: 1008 GB/s on the same 384-bit bus, 10752 cores, 450 W. The fastest single-card token generator of the Ampere era.
The faster GDDR6X (21 Gbps) on the same 384-bit bus is the headline spec, and since token generation is bandwidth-bound, it translates almost directly: a 13B Q4_K_M at ~42–46 t/s versus ~40 on the 3090; 30B Q4 at ~15–17 t/s versus ~14–16. That's the entire single-card improvement: about one token every two seconds more on 30B, which is real but not transformative. The 24 GB capacity is identical, so the model lineup you can run is identical too.
The bigger compute difference shows up in prompt processing (~1600–1900 t/s on 13B vs ~1300–1600 on the 3090) and large-batch serving. If you're running llama-server with several parallel users or huge prompts, the Ti pulls ahead more noticeably than in single-stream chat.
100 W more than the 3090 for ~8% generation speed. Single-card, that's mostly a thermal/power conversation. In a 2× build it's 900 W of GPU, which starts to bite on PSUs, cabling (dual 16-pin 12VHPWR) and case cooling. Many 2× builds deliberately mix a 3090 + 3090 Ti instead — the split tax is the same, you keep NVLink, and you save 100 W.
The 112.5 GB/s bridge means 2× 3090 Ti pairs scale with the low ~10–15% NVLink tax: 48 GB, ~80 t/s on 13B Q4, ~30–35 t/s on 30B Q4, and 70B Q4_K_M at ~17–20 t/s — the fastest 70B consumer build of the Ampere generation. Four cards (96 GB) push 70B to Q5/Q6 at ~28–33 t/s.
Before the numbers: if your model ships a built-in multi-token-prediction head
(e.g. Qwen 3.5/3.6/3.8, Gemma 4), generation can be made faster with
--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1 — on this card that's worth
+45–47% (measured) on token generation; the community A/B on this card ran 42.0 → 60.9 t/s at n-max 2 (128K context, Q4_K_M 27B). It costs ~0.6–2 GB extra VRAM and
does not change output quality, and it only applies to MTP-capable models — Llama-class
models are unaffected. Details and the tuning rules: MTP guide →
| Model | Size on GPU | Token gen | Token gen (MTP est.) | Prompt proc. | Fits? |
|---|---|---|---|---|---|
| 8B (Llama 3.1 8B) | ~4.9 GB | ~109–124 t/s | ~158–182 t/s | ~1700–2000 t/s | ✅ Q8_0, 32K ctx |
| 13B Q4_K_M | ~8.5 GB | ~61–68 t/s | ~88–100 t/s | ~1100–1300 t/s | ✅ 32K+ ctx |
| 30B Q4_K_M | ~18 GB | ~22–25 t/s | ~32–37 t/s | ~500–600 t/s | ✅ 8K–16K ctx |
| MoE 30B-A3B (Qwen3-30B) | ~18 GB | ~87–110 t/s | ~126–162 t/s | ~900 t/s | ✅ fast (3B active) |
| 27B Q5_K_M | ~22 GB | ~19–22 t/s | ~28–32 t/s | ~400 t/s | ⚠️ 8K ctx max |
| 70B Q4_K_M | ~41 GB | — | — | — | ❌ (2× territory) |
MTP est. = rough prior with --spec-type draft-mtp for models that ship MTP heads (Qwen 3.5/3.6/3.8, Gemma 4); Llama-class models get no MTP speedup. Actual gains depend on model, quant, context length, llama.cpp build and your card — sweep --spec-draft-n-max, don't trust the column blindly. MTP guide →
Estimates, full GPU offload, ~8K context.
| Quant | GGUF size | 1× 3090 Ti (24 GB) | 2× 3090 Ti (48 GB) |
|---|---|---|---|
| Q4_K_M | ~19 GB | ✅ ~15–17 t/s, 8–16K ctx | ✅ (overkill, ~32 t/s) |
| Q5_K_M | ~22 GB | ⚠️ ~13–15 t/s, 8K ctx max | ✅ ~26–29 t/s, 16K ctx |
| Q6_K | ~25 GB | ❌ ~1 GB over | ✅ ~24–27 t/s, 16K ctx |
| Q8_0 | ~33 GB | ❌ | ✅ ~19–22 t/s, 8–16K ctx |
Same 24 GB wall as the 3090 — Q4 comfortable, Q5 tight, Q6 one GB short — with ~8% faster token generation across the board thanks to the 1008 GB/s bus. The dual build is the fastest Ampere 27B rig: Q6_K at ~25 t/s is the configuration to want — near-full-fidelity weights at a speed where you stop noticing you're on a local box. Q8 fits too if you want the last few percent of quantization quality.
-c 16384 is the reference config; Q5 stays at -c 8192.--cache-type-k q8_0) to buy context length on the 24 GB single-card builds.Single-card numbers above (27B Q4/Q5 already fit). The 3090 Ti is the 3090 with ~8% more bandwidth and 100 W more heat — the multi-card tables below are the 3090's, plus that ~8%, plus the power bill.
| Build | GPU power | VRAM | 27B Q6_K | 27B Q8_0 | 70B Q4_K_M | 70B Q5–Q6 / Q8 |
|---|---|---|---|---|---|---|
| 2× 3090 Ti (NVLink) | 900 W | 48 GB | ~24–27 t/s | ~19–22 t/s | ~17–20 t/s | ~14–16 t/s |
| 3× 3090 Ti (NVLink+PCIe) | 1350 W | 72 GB | ~29 t/s | ~25–28 t/s | ~21–24 t/s | ~17–19 t/s, 32K+ ctx |
| 4× 3090 Ti (2× NVLink) | 1800 W | 96 GB | ~35 t/s | ~32–36 t/s | ~26–30 t/s | Q5–Q6 ~27–30 · Q8 ~16–18 |
4× 3090 Ti is the endgame Ampere rig: 96 GB, and yes — a 70B at Q8_0 (~70 GB) fits and runs at ~16–18 t/s. That configuration exists on exactly one build class in the RTX 30 Series. 3× (72 GB) is the long-context 70B point. Same NVLink caveat as the 3090: 2-way bridges only, so 3×/4× scale between PCIe and NVLink rates. Hardware: WRX80/server board, x8-spaced slots for 3-slot cards, 2000 W+ PSU, and serious airflow for 1.8 kW of GPU.
When a model doesn't fit, -ngl N keeps the last layers on the CPU instead.
The rule to internalize: offloaded layers run at CPU speed, not a percentage of GPU
speed. A 27B model that's half offloaded is not 50% of GPU speed — it's ~3–5 t/s
total on a modern 8-core. Practical patterns: drop just the last 2–4 layers to the CPU to
reclaim ~0.5–1 GB of VRAM (nearly free); keep MoE experts on CPU with --n-cpu-moe;
park the output embedding with -ot output=CPU to save another ~0.5 GB on big
models. Budget 32 GB of system RAM for 27B-class offload, 64 GB for 70B. Never offload the
KV cache — the conversation gets unusably slow. Same story as the 3090: offload is a fine-tuning tool here, not a strategy — the main use is -ot output=CPU to save ~0.5–1 GB for context on the 70B builds.
llama-server -m 70b-q4_k_m.gguf -ngl 99 -sm layer -c 32768 --tensor-split 1,1,1llama-server -m 70b-q8_0.gguf -ngl 99 -sm layer -c 8192 --tensor-split 1,1,1,1 -ot output=CPU
llama-server -m qwen3-30b-a3b-q4_k_m.gguf -ngl 99 -c 32768 -b 2048 -ub 2048 --host 127.0.0.1 --port 8080
The fastest Ampere single card for LLMs — by a thin, 450-W margin. Everything the 3090 does, it does ~8% faster on token generation, with a bigger prompt-processing edge for parallel serving. For a 2× or 4× NVLink build it's the endgame Ampere config; for a single card, the 3090 does 92% of the work at 78% of the power. The 3090 Ti is the enthusiast's ceiling, not the value pick.