The entry ticket to Blackwell. It's the one 50-series card on GDDR6 — 320 GB/s on a 128-bit bus and 8 GB. A faster 3050, not a different tier.
320 GB/s is the lowest bandwidth in the RTX 50 Series lineup — GDDR6 on a 128-bit bus, comfortably ahead of the 3050's 224 GB/s but a step below the 5060's GDDR7. Token generation re-reads the whole model from VRAM for every token, so speed is roughly proportional to bandwidth: a 5060 (448 GB/s) generates about 40% faster tokens from the same model, and a 5090 (1792 GB/s) ~5.5× faster. Nothing you can configure changes this — it's silicon.
For 7–8B models it's plenty with a good context. The KV cache eats the remainder: at 8K context an 8B model needs roughly 0.5 GB, at 32K roughly 1.5–2 GB. Below 4096 context you can even run Q6/Q8 quants of 8B models. 13B at Q4_K_M (~8.5 GB) loads only with a shrunk context — and it's slow enough that the quality question rarely wins.
2560 CUDA cores with the new Blackwell pipeline keeps prompt processing in the few hundred t/s for small models — perfectly usable for chat, but long documents take a few seconds to digest where a 5090 gobbles them in under a second.
Before the numbers: if your model ships a built-in multi-token-prediction head
(e.g. Qwen 3.5/3.6/3.8, Gemma 4), generation can be made faster with
--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1 — on this card that's worth
+40–70% (estimated) on token generation; no per-card community A/B exists for this card yet; the estimate comes from the low-bandwidth end of the record, which tends to gain the most at n-max 2. It costs ~0.6–2 GB extra VRAM and
does not change output quality, and it only applies to MTP-capable models — Llama-class
models are unaffected. Details and the tuning rules: MTP guide →
| Model | Size on GPU | Token gen | Token gen (MTP est.) | Prompt proc. | Fits? |
|---|---|---|---|---|---|
| 8B (Llama 3.1 8B, Mistral) | ~4.9 GB | ~44–60 t/s | ~62–102 t/s | ~350–450 t/s | ✅ Q8_0, 16K+ ctx |
| 13B Q4_K_M | ~8.5 GB | ~22–33 t/s | ~31–56 t/s | ~300 t/s | ⚠️ tight, short ctx only |
| 13B Q3_K_M | ~6.7 GB | ~25–35 t/s | ~35–63 t/s | ~330 t/s | ✅ with quality loss |
| 27B | ~19 GB | — | — | — | ❌ no |
MTP est. = rough prior with --spec-type draft-mtp for models that ship MTP heads (Qwen 3.5/3.6/3.8, Gemma 4); Llama-class models get no MTP speedup. Actual gains depend on model, quant, context length, llama.cpp build and your card — sweep --spec-draft-n-max, don't trust the column blindly. MTP guide →
Estimates for full GPU offload (-ngl 99), ~8K context, batch 2048. Token generation scales roughly with memory bandwidth; individual runs vary by model architecture (GQA vs MHA, MoE) and llama.cpp build.
| Quant | GGUF size | 1× 5050 (8 GB) | 2× 5050 (16 GB) |
|---|---|---|---|
| Q4_K_M | ~19 GB | ❌ ~11 GB over | ❌ ~3 GB over |
| Q5_K_M | ~22 GB | ❌ | ❌ |
| Q6_K | ~25 GB | ❌ | ❌ |
| Q8_0 | ~33 GB | ❌ | ❌ |
Every 27B quant is out of reach, single or doubled — 27B becomes a thing at 4× (32 GB), where Q4_K_M lands around ~12–16 t/s at 4–8K context. The only single/2-card path is partial CPU offload (~2–4 t/s) — a demo, not a daily driver.
Single-card numbers live in the table above. No NVLink: the tensor split runs over PCIe 5.0 with a ~20–30% scaling tax, and 8 GB per tile caps which models fit.
| Build | GPU power | VRAM | 13B Q4 | 27B Q4_K_M |
|---|---|---|---|---|
| 2× 5050 | 260 W | 16 GB | ~32–42 t/s | ❌ ~3 GB over |
| 4× 5050 | 520 W | 32 GB | ~40–52 t/s | ~12–16 t/s, 4–8K ctx |
Four cards is where 27B Q4 technically fits (32 GB) — at a pace that's honest but modest. Hardware reality: you need a board with four working slots (workstation/server class); the 2-slot 5050 bodies are the only thing that makes 4-way spacing easy in a normal case.
When a model doesn't fit, -ngl N keeps the last layers on the CPU instead.
The rule to internalize: offloaded layers run at CPU speed, not a percentage of GPU
speed. Practical patterns: drop just the last 2–4 layers to the CPU to reclaim ~0.5–1 GB
of VRAM (nearly free); keep MoE experts on CPU with --n-cpu-moe; park the output
embedding with -ot output=CPU to save another ~0.5–1 GB on big models. Budget 32 GB
of system RAM for 27B-class offload, 64 GB for 70B. Never offload the KV cache — the conversation
gets unusably slow.
llama-server -m 13b-q4_k_m.gguf -ngl 99 -sm layer -c 8192 --tensor-split 1,1
llama-server -m llama-3.1-8b-q4_k_m.gguf -ngl 99 -c 8192 -b 2048 -ub 2048 --host 127.0.0.1 --port 8080
A perfectly happy 8B machine and the cheapest way into Blackwell. 130 W is a genuine plus for a 24/7 box, but the 128-bit GDDR6 bus means there's no performance reason to pick it over a cheaper 3060 12G if you already own one. The moment you want 16 GB — 13B at full quality, MoE models, 27B at Q3 — the 5060 Ti is the card that changes your options.