The entry ticket. A 128-bit memory bus caps it, but 8 GB of VRAM means small models still run comfortably all on GPU.
224 GB/s is the lowest bandwidth in the RTX 30 Series lineup, and it shows directly in token generation speed. Because token generation re-reads the whole model from VRAM for every token, speed is roughly proportional to bandwidth: a 3060 12GB (360 GB/s) generates about 60% faster tokens from the same model, and a 3090 (936 GB/s) about 4× faster. Nothing you can configure changes this — it's silicon.
Eight GB sounds small next to the 24 GB monsters, but for 7–8B models it's plenty with a good context. The KV cache is what eats the remainder: at 8K context an 8B model needs roughly 0.5 GB, at 32K roughly 1.5–2 GB. Below 4096 context you can even run Q5/Q6 quants of 8B models.
2560 CUDA cores is a modest count. Prompt processing (first-token latency) lands in the few hundred t/s for small models — perfectly usable for chat, but long documents will take a few seconds to digest where a 3090 gobbles them in under a second.
Before the numbers: if your model ships a built-in multi-token-prediction head
(e.g. Qwen 3.5/3.6/3.8, Gemma 4), generation can be made faster with
--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1 — on this card that's worth
+40–70% (estimated) on token generation; no large community A/B exists for this card yet; the estimate comes from lower-bandwidth cards in the community record, which tend to gain the most at n-max 2. It costs ~0.6–2 GB extra VRAM and
does not change output quality, and it only applies to MTP-capable models — Llama-class
models are unaffected. Details and the tuning rules: MTP guide →
| Model | Size on GPU | Token gen | Token gen (MTP est.) | Prompt proc. | Fits? |
|---|---|---|---|---|---|
| 8B (Llama 3.1 8B, Mistral) | ~4.9 GB | ~30–42 t/s | ~42–71 t/s | ~300–400 t/s | ✅ comfortable, 16K+ ctx |
| 13B (Llama 3 13B) | ~8.5 GB | ~16–24 t/s | ~22–41 t/s | ~200 t/s | ⚠️ tight, short ctx only |
| 13B Q3_K_M | ~6.7 GB | ~18–26 t/s | ~25–44 t/s | ~230 t/s | ✅ with quality loss |
| 30B | ~18 GB | — | — | — | ❌ no |
MTP est. = rough prior with --spec-type draft-mtp for models that ship MTP heads (Qwen 3.5/3.6/3.8, Gemma 4); Llama-class models get no MTP speedup. Actual gains depend on model, quant, context length, llama.cpp build and your card — sweep --spec-draft-n-max, don't trust the column blindly. MTP guide →
Estimates for full GPU offload (-ngl 99), ~8K context, batch 2048. Individual runs vary by model architecture (GQA vs MHA, MoE) and llama.cpp build.
27–32B-class models (Qwen3-32B, Qwen2.5-32B, Gemma 3 27B) are the common "one size up" target from 13B. Their GGUF weights look like this:
| Quant | GGUF size | 1× 3050 (8 GB) | 2× 3050 (16 GB) |
|---|---|---|---|
| Q4_K_M | ~19 GB | ❌ ~11 GB over | ❌ ~3 GB over |
| Q5_K_M | ~22 GB | ❌ | ❌ |
| Q6_K | ~25 GB | ❌ | ❌ |
| Q8_0 | ~33 GB | ❌ | ❌ |
Every 27B quant is out of reach, single or doubled. The only path is partial CPU
offload: load the ~12 transformer layers that fit in 8 GB onto the GPU
(-ngl 12) and let the rest stream from system RAM. That lands around
~2–4 t/s on Q4_K_M — it works, it's honest, and it's slower than most people
want to live with. Treat 27B on a 3050 as a demo, not a daily driver.
Single-card 3050 numbers live in the performance table above; this section is about what 2–4 cards unlock. Three rules: bandwidths sum, there's no NVLink so the tensor split runs over PCIe 4.0 with a ~30% scaling tax, and 8 GB per tile caps which models fit.
| Build | GPU power | VRAM | 13B Q4 | 27B Q4_K_M | 27B Q5_K_M | 27B Q6/Q8 |
|---|---|---|---|---|---|---|
| 2× 3050 | 248 W | 16 GB | ~19 t/s | ❌ ~3 GB over | ❌ | ❌ |
| 3× 3050 | 372 W | 24 GB | ~20 t/s | ⚠️ ~8–9 t/s, 4–8K ctx | ❌ | ❌ |
| 4× 3050 | 496 W | 32 GB | ~24 t/s | ~10–12 t/s, 8K ctx | ⚠️ ~9 t/s, 4–8K ctx | ❌ |
Three cards is where a 27B Q4_K_M technically fits (24 GB) — and it's slow enough that "technically" is doing heavy lifting. The 4-card build (32 GB) is the first one that runs 27B Q4 at a usable pace. Hardware reality: you need a board with four working x8/x16 slots — consumer ATX rarely, workstation and server boards do — and the 3050's 2-slot bodies are the only thing that makes 4-way spacing easy in a normal case.
When a model doesn't fit, -ngl N keeps the last layers on the CPU instead.
The rule to internalize: offloaded layers run at CPU speed, not a percentage of GPU
speed. A 27B model that's half offloaded is not 50% of GPU speed — it's ~3–5 t/s
total on a modern 8-core. Practical patterns: drop just the last 2–4 layers to the CPU to
reclaim ~0.5–1 GB of VRAM (nearly free); keep MoE experts on CPU with --n-cpu-moe;
park the output embedding with -ot output=CPU to save another ~0.5 GB on big
models. Budget 32 GB of system RAM for 27B-class offload, 64 GB for 70B. Never offload the
KV cache — the conversation gets unusably slow. With 8 GB per tile, offloading is your most-used tool on this build — e.g. 27B Q4 on 4× 3050 with --n-cpu-moe-free dense models runs best with -c 4096 and the last 2–3 layers on CPU.
llama-server -m 27b-q4_k_m.gguf -ngl 99 -sm layer -c 8192 --tensor-split 1,1,1llama-server -m 27b-q5_k_m.gguf -ngl 99 -sm layer -c 4096 --tensor-split 1,1,1,1
llama-server -m llama-3.1-8b-q4_k_m.gguf -ngl 99 -c 8192 -b 2048 -ub 2048 --host 127.0.0.1 --port 8080
The RTX 3050 is a perfectly happy 8B machine. Buy it or own it for LLMs only if your ceiling is small models and a modest power bill (130 W is a genuine plus). The moment you want 13B+ at full quality, the 3060 12GB is the upgrade that actually changes your options — and it is usually cheaper than the 3050 on the used market.