Guide: Multi-Token Prediction (MTP)

What MTP is

Standard generation reads the model weights once per token — bandwidth-bound, per the inference basics. Multi-token prediction works around that: the model ships with auxiliary prediction heads that are trained (by the MTP objective) to predict upcoming tokens. At inference:

1. Draft — the MTP head proposes the next 1–N candidate tokens from the backbone's hidden state.
2. Verify — the main backbone runs one forward pass that produces logits for all draft positions in parallel.
3. Accept or reject — drafts that match the main model's distribution are committed for free; the first mismatch falls back to normal sampling from the verified logits.

Because verification is exact (rejection sampling preserves the model's output distribution), there is no quality trade-off — MTP is pure throughput. One weight read yields multiple committed tokens instead of exactly one. That trades idle compute — which bandwidth-bound generation leaves sitting around — against fewer full forward passes.

Which models have MTP

MTP only works when the checkpoint carries the MTP heads. As of writing:

• Qwen 3.5 / 3.6 / 3.8 family — MTP trained inside the model. Qwen3.8 GGUFs ship the heads inline as blk.*.nextn.* tensors — nothing to download separately. (Qwen 3.5/3.6 also get dedicated -MTP GGUF uploads.)
• Gemma 4 — MTP trained separately by Google and released as assistant variants; unsloth's GGUF packages embed the MTP file inside the normal download, so regular Gemma 4 GGUFs just work.

Classic models (Llama, Mistral, older Qwens) have no MTP heads — the flag does nothing for them. When a card page on this site shows an "MTP est." column, it's describing MTP-capable models; your Llama-3.1-8B won't see any of those numbers.

The flag

• --spec-type draft-mtp — use the built-in MTP head as the draft model.
• --spec-draft-n-max N — how many tokens to draft per step. Start at 2; it's not universally optimal (see the rules below).
• --spec-draft-p-min 0.60–0.75 — optional confidence gate: only verify drafts the head is fairly sure about. Big help on bandwidth-poor hardware, harmful on the fastest cards. Sweep it, don't assume it.
• --parallel 1 — for measuring (and often for single-user serving): the spec advantage is a single-stream optimization and is gone by 4 concurrent streams.

llama.cpp's MTP support is young and improving fast — a current build alone was worth +10–15% in some community A/B tests before any flag was touched. Rebuild before you tune.

What to expect: reported numbers

The community A/B record behind these numbers (Qwen3.8-27B, probe.py streaming client, median of 3×3 runs, both arms at --parallel 1) spans 50+ configs from 2016 Pascal to desktop Blackwell, plus AMD, Apple Silicon and APUs. The shape of the data:

The pattern matches the site's core thesis: bandwidth-starved cards get large % gains at n-max 2 (the drafter turns idle cycles into tokens), while the highest-bandwidth cards need deeper drafts (n-max 4–7) to squeeze the biggest absolute jumps out of their compute headroom. And on genuinely bandwidth-limited metal (base M4, low-clock iGPUs) the gain can vanish — the verify pass has no free lunch to take. Deltas are the durable part; absolute numbers move with builds, drivers and thermals.

The seven community rules

Mapped by 40 contributors in the first week after Qwen3.8 shipped. In order of how often they bite:

1. n-max is card- and topology-dependent. 24 GB-class cards peak at 2; bigger or faster cards at 3–4 (some 5090s at 6–7). Changing anything — split mode, quant, context — moves the optimum, so re-sweep after any config change.
2. The confidence gate helps starved cards and hurts fast ones. --spec-draft-p-min 0.60 made deep drafting nearly free on bandwidth-poor rigs; on desktop Blackwell it lowered throughput while raising acceptance. Acceptance is a vanity metric — sweep the gate, keep whichever is faster.
3. The gain scales with generation length. Short answers can cost more than they earn; on already-bandwidth-bound rigs the full gain shows by ~400 tokens and length adds little beyond that.
4. On multi-GPU boxes, fix the split mode before touching spec flags. Default --split-mode layer serializes decode; tensor was +68% on its own on one 5060 Ti pair, and the two levers stack (3.14× combined).
5. Speculation is single-stream. The advantage is gone by --parallel 4. And a --parallel 2 baseline reads ~20% low, inflating your gain claim — always A/B at --parallel 1.
6. Rebuild llama.cpp, and re-derive flags per card class. Upstream optimizes this path weekly (one contributor's fresh-build baseline equaled the day-one with-flag number). A config that pays on a bandwidth-starved APU inverted below baseline on a 960 GB/s card.
7. A shared desktop halves everything, silently. A live compositor + browser spilled 3.5 GB of weights to host RAM over PCIe while /health stayed green and decode simply halved. Bench headless, or verify the weights are resident before trusting a number.

Costs and caveats

• VRAM: +0.6–2 GB for the MTP head and draft KV, growing with n-max. The with-MTP footprint — not the baseline — sets your context ceiling. On a packed 10–12 GB card this is what decides whether MTP fits at 32K at all (one 3080 10GB A/B inverted to −46% at 32K for exactly this reason, with 16K as its working point).
• Prompt processing takes a small hit from device-to-host embedding transfers during verification. TTFT lovers: check both numbers.
• Model must ship MTP heads (see above). No heads, no MTP — full stop.
• Quality is unchanged for greedy/sampling parity in the verified path, but treat MTP as a throughput feature of the current software stack: kernels are young, behavior improves (and occasionally shifts) with each llama.cpp release.
• Long context still pays: at 262K context a 5090 posted the record gain in the table — MTP and long-context serving are friends, not rivals.

Measure it yourself

The whole method is two server runs:

The card pages on this site list an "MTP est." column in their performance tables. Treat those as priors to confirm with your own A/B — model, quant, context, llama.cpp build and card all move the number.