Inference Basics — rtxinference.com Guide

The two phases of a request

Every request to a local LLM has two phases, and they stress the GPU in completely different ways. Understanding this is the key to reading every spec sheet on this site.

Phase 1 — Prompt processing (TTFT)

Your whole prompt is processed in parallel to produce the first token. This is a massive matrix-multiply workload: it is compute-bound. What matters: CUDA core count, clock speed, and FP16/INT8 tensor performance. Measured as t/s prompt-processing throughput (often several hundred to a few thousand). This is time to first token (TTFT) — the "how long until it starts typing" number.

Phase 2 — Token generation (the loop you feel)

After the first token, the model generates one token at a time, re-reading the model weights and the growing KV cache from VRAM for every single token. This is memory-bandwidth-bound. What matters almost exclusively: VRAM bandwidth (GB/s) and VRAM capacity. This is the number on every card page: tokens per second of generation — the typing speed you actually experience.

The two numbers that decide everything VRAM size decides which models and quantizations you can run. Memory bandwidth decides how fast they run. A faster card with less VRAM is useless for a model that doesn't fit — a 3090 running a 70B at 15 t/s beats a 5070 that can't load it at all.

Why bandwidth dominates generation

During generation the batch size is 1, so the GPU isn't doing math at its peak — it's streaming. Every token requires reading essentially the whole model (or, for MoE, the active parameters) from VRAM. The arithmetic intensity is so low that even the weakest GPU could do the math instantly; the memory bus is the bottleneck.

Speed ≈ bandwidth / bytes-per-token. A 4.9 GB Q4 8B model on a 360 GB/s 3060 ≈ 360/4.9 ≈ theoretical ~73 t/s; real-world ~35 t/s after overheads and the KV cache reads. Double the bandwidth, roughly double the t/s on the same model.
CUDA cores barely matter for generation. A 4060 (3072 cores) and a 4060 Ti (4352 cores) generate tokens at nearly the same speed — same 128-bit bus, same ~288 GB/s. Compute cores only decide how fast the prompt gets processed.
MoE models break the rule a little. Only active experts are read per token, so an MoE 30B-A3B on a midrange card can outpace a dense 30B on the same card. Total VRAM still has to fit all experts though.

What prompt-processing speed buys you

Compute (CUDA cores × clock × tensor-core efficiency) determines TTFT. It matters when:

You use long prompts (RAG, pasted documents) — a 2000-token prompt at 200 t/s is a 10 s wait; at 1800 t/s it's 1.1 s.
You run long-context sessions — re-processing history on every turn compounds.
You use speculative decoding or big batches — both are compute-hungry.

For normal chat with short prompts, prompt-processing speed is a luxury; for agentic/RAG workloads it becomes the primary spec. This is why a compute-heavy card like the RTX 4090 can feel "faster" in RAG pipelines even where its generation speed equals a 3090 Ti's.

Quick start with llama.cpp

The server mode is what most people want: a local OpenAI-compatible API.

Typical invocation llama-server -m model-q4_k_m.gguf -ngl 99 -c 8192 --host 127.0.0.1 --port 8080

-ngl 99 → offload all layers to GPU
-c 8192 → context length (drives KV cache size)
-sm layer → split mode for multi-GPU
-ngl < total → partial offload when the model doesn't fit
--tensor-split 24,0 → weight per-GPU in a mixed build

Then measure your own numbers with llama-bench -m model-q4_k_m.gguf -ngl 99 — prompt-processing and generation t/s are reported separately, which is exactly the split described above.

Guide: Inference basics

The two phases of a request

Why bandwidth dominates generation

What prompt-processing speed buys you

Quick start with llama.cpp