How much VRAM do you need to run a 70B model locally?

A 70B model needs about 44.5 GiB of GPU memory at 4-bit and about 75.7 GiB at 8-bit, with an 8K context. Those are our estimates for Llama 3.3 70B Instruct at Q4_K_M and Q8_0; with a 32K context they rise to 52.0 GiB and 83.2 GiB. No single consumer graphics card is big enough: even the GeForce RTX 5090, with 32 GB, falls short at 4-bit. The 4-bit model with an 8K context fits a 48 GB workstation card, such as the RTX 6000 Ada Generation at an estimated 15 tok/s, or a 64 GB Mac (about 9.2 tok/s on an M5 Max). For 8-bit weights you need 96 GB-class hardware: the RTX PRO 6000 Blackwell Workstation Edition (96 GB) runs Q8_0 with a 32K context at about 16 tok/s, a Mac Studio with M5 Ultra and 256 GB at about 10 tok/s, and a DGX Spark with 128 GB holds it too, at only about 2.2 tok/s. Every figure here comes from our hardware and model data and our published estimation method; they are estimates, not benchmarks. You can check any other setup in the calculator.

How much memory a 70B model needs

Llama 3.3 70B Instruct is a dense model: all 70B parameters take part in every token. Its memory has three parts:

• Weights: about 39.5 GiB at Q4_K_M (4.85 bits per weight) and 69.3 GiB at Q8_0 (8.5 bits). Our quantization guide explains the formats.
• KV cache: the attention keys and values for every token in the context. Llama 3.3 70B keeps a full cache in all of its layers, so the cache grows in step with the context: about 2.5 GiB at 8K, 10.0 GiB at 32K and 40.0 GiB at its full 131,072-token context, in the default 16-bit format.
• Runtime overhead: CUDA or Metal buffers and activations, which our method adds as a share of the weights.

Estimates from our methodology, using data as of 2026-10-16. Real speed depends on the runtime, drivers and prompt length.

Two things follow. FP16 is not a realistic local option for a 70B model; almost everyone runs it quantized. And context is the part you control at run time: the same weights need very different hardware at 8K and at 128K.

Which hardware can run a 70B model

For each device, the tables show how much memory the GPU can use and the estimated generation speed at each setting, or "Doesn't fit". Graphics cards first, with NVIDIA's DGX Station at the end:

Estimates from our methodology, using data as of 2026-10-16. Real speed depends on the runtime, drivers and prompt length.

Then Macs and unified-memory mini PCs, where the GPU shares the system's memory:

Estimates from our methodology, using data as of 2026-10-16. Real speed depends on the runtime, drivers and prompt length.

Reading the tables:

• Q4_K_M with an 8K context (44.5 GiB) is the entry point. It fits 48 GB cards with little room to spare: the RTX PRO 5000 48GB Blackwell runs it at about 21 tok/s and the older RTX A6000 at about 12 tok/s. It also fits 64 GB Macs and mini PCs.
• Q4_K_M with a 32K context (52.0 GiB) needs more: the RTX PRO 5000 72GB Blackwell (about 21 tok/s), a 96 GB Mac Studio with M5 Ultra (about 18 tok/s) or a 128 GB machine.
• Q8_0 (75.7 GiB at 8K) needs a 96 GB card or 128 GB or more of unified memory. A 96 GB Mac is not enough: its GPU gets about 72 GB by default.
• Unified memory is not all GPU memory. macOS and the mini PCs' operating systems give the GPU only part of the memory by default (75% on a Mac in our estimates; the Usable column shows each device), so a 64 GB machine has about 48 GB for the model. On a Mac you can raise the limit.
• Speed follows bandwidth, not memory size. The same Q4_K_M model is estimated at 3.8 tok/s on a DGX Spark (273 GB/s), 18 tok/s on an M5 Ultra (1,200 GB/s) and 28 tok/s on an RTX PRO 6000 (1,792 GB/s). Our GPU guide explains why.

Two consumer GPUs instead of one big card

The usual way to run a 70B model on consumer hardware is two 24 GB cards, such as two second-hand GeForce RTX 3090 cards. When a model doesn't fit on one GPU, Ollama spreads it across all the GPUs it finds, and llama.cpp does the same. Two 24 GB cards give 48 GB, enough on paper for Q4_K_M with an 8K context. Three things to know:

• It's tight. Each card carries its own runtime buffers, so the usable total is a little under 48 GB. Keep the context short or quantize the KV cache (see below).
• It adds memory, not speed. With the layers split between the cards, each token passes through the first card's layers and then the second's, so one user gets roughly the speed of a single card that could hold the whole model. Serving engines such as vLLM can split every layer across the cards (tensor parallelism), which raises throughput mainly when many requests run at once. Our GPU guide covers this in more detail.
• Power and space. Each RTX 3090 is rated at 350 W, so two need a large power supply, a motherboard with two well-spaced slots and good airflow.

Our estimator models one device at a time, which is why the tables above have no multi-GPU rows.

Making a 70B model fit in less memory

• Set the context you actually need. Ollama's documentation gives its default context as 4,096 tokens. To use more, set OLLAMA_CONTEXT_LENGTH when you start the server, or /set parameter num_ctx inside ollama run. Remember that every extra token of context costs memory.

OLLAMA_CONTEXT_LENGTH=32768 ollama serve
ollama run llama3.3:70b

• Quantize the KV cache. With Flash Attention on, OLLAMA_KV_CACHE_TYPE=q8_0 stores the cache in about half the memory of the default f16, which Ollama's documentation describes as a very small loss in precision. For Llama 3.3 70B at 32K, that saves about half of the 10.0 GiB cache.
• Go below 4-bit only if you must. Q3_K and the 2–3-bit IQ quants shrink the weights further, at a cost in quality that grows quickly below 4 bits. Our estimates start at Q4_K_M; the quantization guide covers the trade-off, and its worked example shows why Q4_K_M is the highest level that fits a 64 GB Mac.
• Offload to system RAM as a last resort. llama.cpp and Ollama can run the layers that don't fit on the CPU, and ollama ps shows the CPU/GPU split. It works, but system RAM is much slower than GPU memory, so the whole model slows down. Our GPU guide explains why.

Dense or mixture-of-experts?

Not every large model costs the same to run. A mixture-of-experts (MoE) model needs memory for all of its parameters but reads only the active ones for each token. Qwen3-Coder-Next (80B-A3B), with 80B (3B active) parameters, needs about 48.1 GiB at Q4_K_M with an 8K context, close to Llama 3.3 70B, but it is estimated to run many times faster on the same hardware:

Estimates from our methodology, using data as of 2026-10-16. Real speed depends on the runtime, drivers and prompt length.

Treat MoE speeds as upper bounds: routing between experts adds work that our bandwidth formula does not count. Even so, this is why the large-memory, lower-bandwidth machines suit MoE models better than dense 70B ones.

Which machine to buy for a 70B model

• Lowest cost, more effort: two 24 GB cards in a desktop, for Q4_K_M with short contexts. Check your power supply, case and cooling first.
• One card, highest speed: the RTX PRO 6000 Blackwell Workstation Edition (96 GB) fits every setting in the tables, at workstation prices. The 72 GB RTX PRO 5000 covers 4-bit up to a 32K context, and the 48 GB one 4-bit with a short context.
• One quiet box with lots of memory: a Mac Studio. With M5 Ultra, 96 GB covers 4-bit up to a 32K context. For 8-bit you need 128 GB on an M5 Max (about 5.3 tok/s) or 256 GB on an M5 Ultra (about 10 tok/s). You choose the memory when you order, and it can't be added later.
• NVIDIA's software stack in a small box: the DGX Spark with 128 GB holds even Q8_0, but it is slow on dense 70B models. It is a better fit for MoE models.

For prices and value scores side by side, see Mac Studio vs NVIDIA GPU and our price tracker.

OnPrem World sells some of the hardware in these tables by quote, including NVIDIA workstation cards and the DGX Spark and DGX Station. Every device is estimated with the same data and method, whether we sell it or not; see our independence policy.