Choosing a GPU for local LLMs: VRAM vs memory bandwidth

Buy VRAM first and bandwidth second. The amount of video memory (VRAM) on a graphics card decides which models you can load at all: a model that does not fit either fails to load or spills into system RAM and slows to a crawl. Once a model fits, memory bandwidth decides how fast it answers, because generating every new token means reading all of the model's active weights from memory. In practice that makes 16 GB the sensible floor for a new card, 24–32 GB the range for today's popular 24–32B models, and 48 GB or more (or a large unified-memory machine) the entry ticket for 70B-class models. The tables below come from our hardware and model data and our published estimation method; you can check any other pairing in the calculator.

How much memory a model needs

A model in memory has three parts:

• Weights: the number of parameters times the bits stored per weight. Quantization lowers the bits per weight; see our quantization guide.
• KV cache: the attention keys and values for every token in the conversation. It grows with the context length you set.
• Runtime overhead: CUDA or Metal context, scratch buffers and activations.

Take Qwen3-32B, a popular dense model with 32.8B parameters. At Q4_K_M its weights alone take about 18.5 GiB. Add the KV cache and overhead and the estimate is 21.9 GiB at an 8K context:

Estimates from our methodology, using data as of 2026-10-14. Real speed depends on the runtime, drivers and prompt length.

Two things stand out. First, the jump from Q4_K_M to FP16 roughly triples the memory for the same model, which is why almost everyone runs quantized weights locally. Second, context is not free: the KV cache alone grows from 2.0 GiB at 8K tokens to 8.0 GiB at 32K. Newer models with hybrid attention layouts, such as Qwen3.5-27B or Gemma 3 27B IT, keep a full KV cache in only some of their layers, so their memory grows much more slowly with context. Each model page lists its own KV-cache layout.

Leave headroom. Your desktop, browser and other apps also use VRAM, and a model that fits with zero margin will often fail to load or fall back to the CPU. If a model needs 15 GB, a 16 GB card is tight; a 20 or 24 GB card is comfortable.

What fits on 12, 16, 24 and 32 GB

These tables show the highest-quality quantization (up to Q8_0) that fits each card with an 8K context, plus the estimated generation speed. They are estimates, not benchmarks, and they assume the whole model sits in VRAM.

12 GB: GeForce RTX 3060 (12 GB)

Estimates from our methodology, using data as of 2026-10-14. Real speed depends on the runtime, drivers and prompt length.

A 12 GB card runs 7–14B models comfortably, often at Q8_0, which is close to the original quality. The step up to 20B+ is where it runs out.

16 GB: GeForce RTX 5060 Ti 16GB

Estimates from our methodology, using data as of 2026-10-14. Real speed depends on the runtime, drivers and prompt length.

16 GB is the point where mixture-of-experts models such as gpt-oss-20b start to fit, and 24B dense models squeeze in at Q4_K_M with little room to spare.

24 GB: GeForce RTX 4090

Estimates from our methodology, using data as of 2026-10-14. Real speed depends on the runtime, drivers and prompt length.

24 GB is the sweet spot for single-GPU local AI in 2026: every popular 24–32B model fits at Q4_K_M or better. Llama 3.3 70B does not; it needs about 44.5 GiB even at Q4_K_M.

32 GB: GeForce RTX 5090

Estimates from our methodology, using data as of 2026-10-14. Real speed depends on the runtime, drivers and prompt length.

The extra 8 GB over a 24 GB card buys higher-quality quantizations and longer contexts for the same 27–32B models, not a new class of model. 70B dense models and the large MoE models still need more memory.

Why bandwidth decides speed

When a model writes its answer, it produces one token at a time, and each token requires reading every active weight from memory once. The GPU's arithmetic units are mostly waiting for data, so the speed limit is memory bandwidth:

tokens per second ≈ memory bandwidth × efficiency ÷ bytes of active weights read per token

Our methodology assumes discrete GPUs reach 70% of their peak bandwidth. Here is the same small model, Qwen3-8B at Q4_K_M, across cards that can all hold it:

Estimates from our methodology, using data as of 2026-10-14. Real speed depends on the runtime, drivers and prompt length.

The ordering follows bandwidth almost exactly, not the card's age or its gaming performance. The GeForce RTX 4060 Ti 16GB is the clearest example: it has more VRAM than the RTX 3060 12GB but less bandwidth (288 GB/s against 360 GB/s), so a model that fits both is estimated to run faster on the older card.

Two caveats. First, the formula covers generation only. Reading your prompt (prefill) is limited by compute instead, so newer cards with more compute process long documents faster even at similar bandwidth. Second, real numbers depend on the runtime, the driver, the quantization kernel and the prompt length; treat the table as a ranking and an order of magnitude, not a promise.

MoE models: memory by total, speed by active

Mixture-of-experts (MoE) models change the trade-off. All the experts must sit in memory, so memory follows the total parameter count; but each token only runs through a few experts, so speed follows the much smaller active count. gpt-oss-20b has 21B (3.6B active) parameters; Qwen3-30B-A3B (2507) has 30.5B (3.3B active).

That is why MoE models show very high estimated speeds in the tables above. Treat those as upper bounds: routing between experts adds work that the bandwidth formula does not count. Even so, an MoE model that fits your memory will usually answer far faster than a dense model of the same total size, which makes MoE the best use of a large-memory, lower-bandwidth machine.

When a model does not fit: offloading to system RAM

llama.cpp and Ollama can keep some layers on the GPU and run the rest on the CPU from system RAM. llama.cpp controls this with the -ngl (--n-gpu-layers) option, and in Ollama the ollama ps command shows the split in its PROCESSOR column, for example 48%/52% CPU/GPU. It works, and it is a reasonable way to try a model that is slightly too big. But system RAM has a small fraction of a modern GPU's bandwidth, so every layer you move off the card drags the whole model down toward CPU speed. If you find yourself offloading all the time, the model is too big for the card.

Comparing the cards

Launch specs from our database. The last column divides the launch price by the VRAM, which is a quick way to see what a gigabyte cost at release; actual street prices change constantly.

And the workstation cards, which trade price for capacity:

A few patterns:

• Used 24 GB cards. The GeForce RTX 3090 pairs 24 GB with 936 GB/s, close to an RTX 4090 for generation, and it is widely available second-hand. Check its condition and your power supply before buying.
• Power. High-end cards draw a lot: the RTX 5090 is rated at 575 W and the RTX 4090 at 450 W. Follow the card maker's power-supply recommendation.
• More memory per card. Workstation cards such as the RTX PRO 6000 Blackwell Workstation Edition (96 GB) run 70B dense models and large MoE models on one card, at workstation prices.

NVIDIA, AMD or Intel?

The memory and bandwidth rules apply to every vendor, but software support does not.

• NVIDIA (CUDA) is supported by every major runtime: Ollama, llama.cpp, LM Studio and vLLM. It is the path with the fewest surprises.
• AMD (ROCm or Vulkan) works with Ollama and llama.cpp, but supported cards and operating systems differ by runtime. Ollama, for example, needs the ROCm v7 driver and publishes a list of supported Radeon cards; on cards outside it, its Vulkan backend (on by default on Windows and Linux) is the fallback. Check the runtime's GPU list before you buy.
• Intel Arc (SYCL or Vulkan) runs through llama.cpp's SYCL and Vulkan backends, and Ollama supports Intel GPUs through Vulkan. Support in other tools is narrower, so confirm the one you want to use lists your card.

Our hardware pages list the software stack for each card, and the runtime documentation linked in the sources below is the place to confirm current support.

More than one GPU, or no GPU at all

Two cards add memory, not speed. llama.cpp and Ollama split a model's layers across GPUs, so a pair of 24 GB cards can hold a model that needs about 40 GB; but each token still passes through the layers on one card and then the other, so single-user speed stays near that of one card. Serving engines such as vLLM can split each layer across cards (tensor parallelism), which helps throughput when many requests arrive at once. That is the office-server case, not the typical desktop.

The alternative to a bigger GPU is a different kind of machine. Apple Silicon Macs and unified-memory mini PCs such as the DGX Spark or Ryzen AI Max+ 395 (Strix Halo) systems systems offer up to 128 GB or more of memory that the GPU can use, at much lower bandwidth (273 GB/s and 256 GB/s respectively). They load models no consumer card can, and run large MoE models well, but a dense 70B model on them is slow. Our Mac guide covers that trade-off in detail.

A five-minute buying checklist

1. List the models you want to run and the context length you need. Coding agents and document chat need longer contexts than a quick chat.
2. Look up the memory on each model page, or pick your card in the calculator.
3. Pick the cheapest card with enough VRAM plus headroom. Aim for the model's estimate plus 10–20%.
4. Among cards with enough VRAM, prefer bandwidth. It is the best single predictor of generation speed.
5. Check software support for your runtime and operating system, especially for AMD and Intel cards.