Multi-GPU Builds — rtxinference.com Guide

How llama.cpp splits a model

llama.cpp splits a model's transformer layers across GPUs (-sm layer). Two rules of thumb:

Token generation scales well — bandwidths roughly add, so 2× 3090 ≈ double the t/s of one, and you get 48 GB of VRAM (70B Q4 territory).
Inter-GPU traffic is the tax. Layers on different cards exchange activations every token. Over PCIe that's fine-ish; a fast GPU-to-GPU bridge makes 2-GPU splits much cheaper.

Interconnect	Available on	Link speed	Split tax
NVLink (2-way only)	RTX 3090 / 3090 Ti	112.5 GB/s	~10–15%
PCIe 4.0 x16	RTX 30 / 40 Series (all other cards)	~31 GB/s per dir.	~25–35%
PCIe 5.0 x16	RTX 50 Series	~63 GB/s per dir.	~20–30% (est.)

So 2× 3090 (NVLink) lands around ~85–90% of the sum of single-card speeds on token generation, while 2× 4090 or 2× 5090 (PCIe-only) land around 65–80%. Prompt processing scales nearly linearly in all cases because the per-token activation traffic is a much smaller fraction of that phase.

NVLink is the 3090's hidden spec

The RTX 3090/3090 Ti are the only consumer cards ever shipped with a fast GPU-to-GPU bridge. The RTX 40 and 50 Series dropped it; no bridge, no shortcut. Consequences:

2× 3090 remains the 70B reference build — 48 GB, ~10–15% tax, used prices. 2× 4090 gives the same 48 GB with a bigger tax and (at new prices) more money.
3-way and 4-way 3090 builds are the only 3+/4× consumer LLM rigs where a bridge exists at all: the 3rd card joins over PCIe, and a 4× build has two NVLink pairs. Real scaling lands between the pure-PCIe and pure-NVLink rates.
Anything else pairs over PCIe. 3050–3080 Ti, all 40-series, all 50-series. It works — budget builds like 2–4× 3060 12G are still the cheapest route to 24–48 GB — but expect the ~25–35% tax.

Choosing a build

Best 70B value 2× 3090 (NVLink) — 48 GB, ~16 t/s on 70B Q4, ~700 W. The price/performance point; 2× 3090 Ti buys ~12% more speed for more money and 200 W.

Best budget multi 2× 3060 12G — 24 GB for a fraction of the price. 30B Q4 at ~11 t/s is a legitimate "big model" experience, 340 W total. Scales to 3× (36 GB) and 4× (48 GB).

Worst value build 2× 3070 Ti — 580 W, no NVLink, 16 GB. A single 3090 beats it on capability, speed and power. The small cards only tile well as 3060 12G.

Mixed-generation pairs work too: a 3090 + 5090 (56 GB) pair runs 70B with the faster card doing more layers. Weight the split toward the bigger/faster card with --tensor-split. The cross-series comparison has the build-by-build tables.

CPU offloading — when a model outgrows the VRAM

llama.cpp's -ngl N puts the last N transformer layers on the CPU and the rest on the GPUs. The number to internalize: offloaded layers run at CPU speed (~1–3 t/s for a 27B model on a modern 8-core), so the penalty is not proportional:

Scenario	On CPU	Result
27B Q4 in 24 GB: last ~4 layers on CPU to fit	~6%	~85% of full-GPU speed
27B Q4 in 16 GB: ~half the layers offloaded	~50%	~3–5 t/s — a different machine
70B Q4 in 24 GB: ~2/3 offloaded	~67%	~2–3 t/s — demo territory

Keep it under ~10%: dropping a handful of layers to claw back 0.5–1 GB is nearly free; past ~15–20% the t/s curve falls off a cliff.
RAM budget: offloaded weights live in system RAM — 32 GB for 27B-class partial offload, 64 GB for 70B. Dual-channel helps; the CPU streams weights every token.
MoE trick: --n-cpu-moe moves MoE expert weights to CPU — cheap, because only a few experts are read per token. Great for MoE models in tight VRAM.
Output embedding: -ot output=CPU parks the big final layer (~0.5–1 GB on 27B/70B) on CPU — a free VRAM win with negligible speed cost.
Never offload the KV cache — per-token latency collapses.

Offload example llama-server -m 27b-q4_k_m.gguf -ngl 60 -sm layer -c 8192 --tensor-split 1,1 -ot output=CPU 27B Q4 across 2 GPUs, last 4 layers + output embedding on CPU

Commands that matter

Equal cards, split by layer:
llama-server -m model-q4_k_m.gguf -ngl 99 -sm layer --tensor-split 1,1

Unequal pair (weight toward the bigger card):
llama-server -m model.gguf -ngl 99 -sm layer --tensor-split 60,40

3 cards / 4 cards (3090 builds):
llama-server -m 70b-q4_k_m.gguf -ngl 99 -sm layer -c 32768 --tensor-split 1,1,1
llama-server -m 70b-q5_k_m.gguf -ngl 99 -sm layer -c 32768 --tensor-split 1,1,1,1 -ot output=CPU

Hardware reality for 3–4 cards You need a TRX40/WRX80 (or server) board: 4× 3090 wants x16/x8/x8/x8 or x8×4 with 3-slot spacing — the 3090 is a 3-slot card, so slot placement is the whole design problem. PSU: 1200 W for 3×, 1600–2000 W for 4×, all 12VHPWR. Case: full tower with forced front-to-back airflow; four 3090s at load are ~1.4 kW of heat in one box.

Guide: Multi-GPU builds

How llama.cpp splits a model

NVLink is the 3090's hidden spec

Choosing a build

CPU offloading — when a model outgrows the VRAM

Commands that matter