# Quantization explained: Q4_K_M vs Q8_0 vs FP16, and the memory and quality trade-offs

> OnPrem World · https://onpremworld.com/en/guides/quantization-explained · Updated: 2026-10-14 · CC BY 4.0

**Quantization stores each of a model's weights with fewer bits, so the model takes less memory and runs faster, at a small cost in quality.** The original weights are usually 16-bit (FP16 or BF16). Q8_0 uses about 8.5 bits per weight, roughly half the size, and in published measurements it is practically indistinguishable from the original. Q4_K_M uses about 4.85 bits, under a third of the original size, and loses a little quality that most people never notice in everyday use. Below 4 bits the losses grow quickly. That is why Q4_K_M is the default almost everywhere, and why the usual advice is: run Q4_K_M, step up to Q5_K_M, Q6_K or Q8_0 if you have memory to spare, and only drop below 4-bit when the alternative is not running the model at all. Every memory figure on our [model pages](https://onpremworld.com/en/models.md) and in the [calculator](https://onpremworld.com/en/can-i-run.md) is computed per quantization level.

## What quantization does

A model is billions of numbers. In the files published by model makers, each is usually a 16-bit floating-point value: FP16, or BF16, which keeps float32's exponent range with fewer mantissa bits. Quantization replaces those numbers with low-bit integers plus a few shared scale factors per block of weights. When the model runs, each weight is reconstructed as an integer times its block's scale, which is close to, but not exactly, the original value.

The saving is direct. Weight memory is the number of parameters times the bits per weight, divided by eight:

| Quantization | Bits per weight | [Llama 3.1 8B Instruct](https://onpremworld.com/en/models/llama-3.1-8b.md) | [Qwen3-32B](https://onpremworld.com/en/models/qwen3-32b.md) | [Llama 3.3 70B Instruct](https://onpremworld.com/en/models/llama-3.3-70b.md) |
|---|---:|---:|---:|---:|
| Q4_K_M | 4.85 | 4.5 GiB | 18.5 GiB | 39.5 GiB |
| Q5_K_M | 5.69 | 5.3 GiB | 21.7 GiB | 46.4 GiB |
| Q6_K | 6.59 | 6.1 GiB | 25.2 GiB | 53.7 GiB |
| Q8_0 | 8.5 | 7.9 GiB | 32.5 GiB | 69.3 GiB |
| FP16 | 16 | 14.9 GiB | 61.1 GiB | 130.4 GiB |

*Estimates from our [methodology](https://onpremworld.com/en/methodology.md), using data as of 2026-10-14. Real speed depends on the runtime, drivers and prompt length.*

That table is the difference between a 70B model needing a datacentre GPU and fitting on a 64 GB Mac. The bits per weight are the averages we use in our [methodology](https://onpremworld.com/en/methodology.md); a real file can differ slightly, because each model has a different mix of tensor shapes.

## Reading the names

The names come from llama.cpp's GGUF format, the most common format for local models:

- **Q** means quantized, and the **number** is the main bit width: Q4 is 4-bit, Q8 is 8-bit.
- **K** marks the k-quants scheme introduced in llama.cpp in 2023 (pull request #1684). Weights are grouped into super-blocks, and the scales of each block are quantized too. The Hugging Face GGUF documentation describes the Q4_K block format as working out to 4.5 bits per weight and Q6_K to 6.5625.
- **S, M, L** (small, medium, large) say how many tensors are promoted to a higher bit width. Q4_K_S keeps everything at 4-bit; Q4_K_M stores some of the most sensitive tensors, parts of the attention value and feed-forward down projections, at 6-bit. That is why Q4_K_M averages a little more than 4.5 bits per weight.
- **_0 and _1** (such as Q4_0, Q8_0) are the older, simpler formats. Hugging Face lists them as legacy, but Q8_0 is still the standard 8-bit choice.
- **IQ** types (IQ2_XS, IQ3_M, IQ4_XS and others) use an importance matrix and non-linear grids to keep quality at very low bit widths.
- **F16 / FP16 and BF16** are unquantized 16-bit weights.

## How much quality you lose

llama.cpp publishes careful measurements for Llama 3 8B: perplexity on the Wikitext-2 test set (lower is better), the increase over FP16, and KL divergence, which measures how far the quantized model's predicted probabilities drift from the original's:

| Quantization | Size (GiB) | Perplexity | Increase vs FP16 | KL divergence |
|---|---:|---:|---:|---:|
| FP16 | 14.97 | 6.2332 | – | – |
| Q8_0 | 7.96 | 6.2343 | +0.0027 | 0.0014 |
| Q6_K | 6.14 | 6.2534 | +0.0217 | 0.0055 |
| Q5_K_M | 5.33 | 6.2886 | +0.0570 | 0.0108 |
| Q4_K_M | 4.58 | 6.4071 | +0.1755 | 0.0313 |
| Q4_0 | 4.34 | 6.7001 | +0.4685 | 0.0719 |
| Q3_K_M | 3.74 | 6.8885 | +0.6569 | 0.1019 |
| Q2_K | 2.96 | 9.7516 | +3.5199 | 0.4451 |

*Source: llama.cpp perplexity README, "LLaMA 3 8b Scoreboard" (no importance matrix), measured on an RTX 4090.*

Another way to read it: the same page reports how often the quantized model picks the same most-likely next token as FP16. That was 97.7% for Q8_0, 96.0% for Q6_K, 91.9% for Q4_K_M and 71.1% for Q2_K. The curve is gentle down to about 4 bits and steep below it. Note also that Q4_K_M, at almost the same size as the older Q4_0, loses well under half as much: the K-quant mix matters.

Three caveats keep these numbers honest:

1. **They are for one model.** llama.cpp's own documentation warns that perplexity cannot be compared directly between models, and that Llama 3 lost more to quantization than Llama 2 did.
2. **Bigger models tolerate quantization better.** Qwen's documentation measured its Qwen2 models on MMLU, C-Eval and IFEval. At 72B, 4-bit GPTQ scored 81.2 on average against 81.3 for BF16, essentially unchanged; at 7B, 4-bit dropped from 66.9 to 64.1; at 0.5B, from 34.4 to 29.7.
3. **Perplexity is not the same as usefulness.** Fine detail such as exact numbers, rare languages or long chains of reasoning can suffer before a benchmark average moves. If a task matters, test the quantization you plan to use on it.

The second point leads to a useful rule. With a fixed amount of memory, a larger model at Q4_K_M is often a better choice than a smaller model at Q8_0. In Qwen's data, the 72B model at 4-bit is far ahead of the 7B model at full precision.

## What it does to speed

Generating text is limited by memory bandwidth: every token requires reading all of the active weights. Fewer bits per weight means fewer bytes to read, so lower quantizations generate faster. llama.cpp's quantize README measured Llama 3.1 8B at about 72 tokens per second at Q4_K_M, 51 at Q8_0 and 29 at F16 on the same machine. Our estimates follow the same logic. For [Qwen3-32B](https://onpremworld.com/en/models/qwen3-32b.md) on an [GeForce RTX 5090](https://onpremworld.com/en/hardware/rtx-5090.md): about 63 tokens per second at Q4_K_M, 46 at Q6_K and 36 at Q8_0.

Reading the prompt (prefill) behaves differently, because it is limited by compute rather than bandwidth; quantization helps it less and can even slow it on some hardware.

## Memory needed for a whole model

Weights are not the whole story. The KV cache (which grows with context length) and runtime overhead come on top. For a typical 32B model:

| Quantization | Bits per weight | Memory needed @ 4K | Memory needed @ 8K | Memory needed @ 32K |
|---|---:|---:|---:|---:|
| Q4_K_M | 4.85 | 20.9 GiB | 21.9 GiB | 27.9 GiB |
| Q5_K_M | 5.69 | 24.3 GiB | 25.3 GiB | 31.3 GiB |
| Q6_K | 6.59 | 27.9 GiB | 28.9 GiB | 34.9 GiB |
| Q8_0 | 8.5 | 35.6 GiB | 36.6 GiB | 42.6 GiB |
| FP16 | 16 | 65.6 GiB | 66.6 GiB | 72.6 GiB |

*Estimates from our [methodology](https://onpremworld.com/en/methodology.md), using data as of 2026-10-14. Real speed depends on the runtime, drivers and prompt length.*

On a 24 GB graphics card, that table says Q4_K_M with up to about 8K of context fits, Q5_K_M just misses, and Q8_0 needs a 48 GB class card. Every [model page](https://onpremworld.com/en/models.md) has the same table, and the [calculator](https://onpremworld.com/en/can-i-run.md) checks it against your hardware.

## A worked example: does a 70B model fit a 64 GB Mac?

Putting the pieces together for a common question. [Llama 3.3 70B Instruct](https://onpremworld.com/en/models/llama-3.3-70b.md) has 70B parameters:

1. **Weights**: about 39.5 GiB at Q4_K_M, 69.3 GiB at Q8_0 and 130.4 GiB at FP16.
2. **Add the KV cache and overhead**: with an 8K context, Q4_K_M needs about 44.5 GiB in total.
3. **Compare with usable memory**: a Mac's GPU can use only part of its unified memory by default; our estimates assume 75%, so a 64 GB Mac offers about 64 GB × 75%.

So Q4_K_M just fits and Q5_K_M does not. Whether a machine "can run" a model often comes down to the quantization you choose; how fast it runs once it fits is a question of memory bandwidth, covered in our [Mac guide](https://onpremworld.com/en/guides/mac-local-llm.md).

## Choosing a level

| Your situation | Pick |
|---|---|
| The model only just fits | Q4_K_M, or an IQ4/Q3_K quant with an importance matrix if even that does not fit |
| Normal use, some headroom | Q4_K_M or Q5_K_M |
| Plenty of memory, quality matters (coding, maths, non-English text) | Q6_K or Q8_0 |
| Small models (under about 4B parameters) | Q8_0: small models lose the most to quantization and are cheap to run at 8-bit |
| Making your own quantizations or fine-tuning | Keep the FP16/BF16 original as the source |

Below Q4, check whether a smaller model at Q4_K_M or above would serve you better: the table above shows how steeply quality falls at 3 and 2 bits.

## Importance matrices and IQ quants

An importance matrix (imatrix) is computed by running the model over a sample of text to find which weights affect the output most; the quantizer then spends its limited precision on those weights. In llama.cpp's Llama 3 8B scoreboard, adding an importance matrix reduced Q4_K_M's perplexity increase from about 0.18 to 0.15 at the same file size. The IQ types are built around it and are the best option when you need 2 or 3 bits to fit a model at all.

## The KV cache can be quantized too

The KV cache is stored at 16-bit by default. For long contexts it can take several gigabytes, so runtimes let you quantize it separately from the weights:

- **Ollama**: set `OLLAMA_KV_CACHE_TYPE=q8_0` (about half the memory of the f16 default, with a very small loss) or `q4_0` (about a quarter, with a loss that may be noticeable at long contexts). It applies to all models and needs Flash Attention, which Ollama enables automatically where the hardware supports it.
- **llama.cpp**: `--cache-type-k q8_0 --cache-type-v q8_0` (or `-ctk` and `-ctv`). A quantized V cache requires Flash Attention, which llama.cpp turns on automatically in its default `auto` mode.

Ollama's documentation notes that models with a high grouped-query attention ratio can be more sensitive to KV-cache quantization, so test it on the model you use.

## Picking a quantization in practice

- **Ollama**: the default tag is usually Q4_K_M; other levels are separate tags, for example `ollama run qwen3:8b-q8_0`. See the model's tags page in the Ollama library.
- **LM Studio**: choose the quantization in the download list, or from the command line with `lms get llama-3.1-8b@q4_k_m`.
- **llama.cpp**: add the quantization to the Hugging Face repository name, as in `-hf <user>/<model>:Q6_K`; without it, llama.cpp picks Q4_K_M by default.
- **Hugging Face**: GGUF files carry the quantization in their name, such as `model-Q4_K_M.gguf`.

## Other formats you will meet

- **MLX** (Apple Silicon): `mlx_lm.convert -q` quantizes to 4-bit with a group size of 64 by default. MLX also supports the newer mxfp4, nvfp4 and mxfp8 formats. See our [Mac guide](https://onpremworld.com/en/guides/mac-local-llm.md).
- **GPU servers (vLLM and similar)**: AWQ, GPTQ, FP8 and bitsandbytes are the common formats. FP8 computation needs an NVIDIA GPU with compute capability 8.9 or newer (Ada Lovelace, Hopper, Blackwell); older GPUs from Turing onwards can run FP8 models as weight-only. vLLM's support for GGUF is still described as highly experimental. Our [runtime comparison](https://onpremworld.com/en/guides/ollama-vs-lm-studio-vs-llama-cpp-vs-vllm.md) covers when vLLM is the right tool.

## Frequently asked questions

### Is Q4_K_M good enough?

For most people, yes. In llama.cpp's published measurements on Llama 3 8B, Q4_K_M raised perplexity by about 0.18 over FP16 and picked the same top token about 92% of the time, at under a third of the size. It is the default tag for models such as qwen3:8b in Ollama's library, and llama.cpp picks it by default when downloading from Hugging Face. If you have memory to spare, Q5_K_M or Q6_K get closer to the original.

### What is the difference between Q4_K_M and Q4_K_S?

Both are 4-bit K-quants. Q4_K_S stores every tensor at 4-bit, while Q4_K_M keeps some of the most sensitive tensors (parts of the attention and feed-forward layers) at 6-bit. Q4_K_M is slightly larger and measurably better: on Llama 3 8B its perplexity increase was about 0.18 against 0.27 for Q4_K_S.

### Should I use Q8_0 or FP16?

Q8_0. In llama.cpp's measurements on Llama 3 8B, Q8_0 was practically indistinguishable from FP16 (a perplexity increase of 0.003) at about half the memory. FP16 or BF16 only makes sense when memory is plentiful, for very small models, or as the source for making your own quantizations.

### Does quantization make a model faster?

Usually, yes. Generating text is limited by how many bytes are read from memory per token, so fewer bits per weight means faster generation. By our estimate Qwen3-32B on an RTX 5090 runs at about 63 tokens per second at Q4_K_M against 36 at Q8_0.

### What are IQ quants and the importance matrix?

An importance matrix (imatrix) is computed by running the model over sample text to find which weights matter most; quantization then spends its precision where it counts. The IQ quant types rely on it and give the best quality at very low bit widths, such as 2 or 3 bits. An imatrix also improves the standard K-quants slightly.

### Does quantizing the KV cache hurt quality?

A little. Ollama's documentation describes q8_0 for the KV cache as using about half the memory of the default f16 with a very small loss that usually has no noticeable impact, and q4_0 as using about a quarter with a loss that may be more noticeable at long contexts. It needs Flash Attention, and models with many query heads per KV head can be more sensitive.

## Sources

- https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md
- https://github.com/ggml-org/llama.cpp/blob/master/tools/perplexity/README.md
- https://github.com/ggml-org/llama.cpp/pull/1684
- https://github.com/ggml-org/llama.cpp/blob/master/tools/imatrix/README.md
- https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- https://huggingface.co/docs/hub/gguf
- https://qwen.readthedocs.io/en/latest/getting_started/quantization_benchmark.html
- https://docs.ollama.com/faq
- https://ollama.com/library/qwen3/tags
- https://lmstudio.ai/docs/cli/local-models/get
- https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.quantize.html
- https://docs.vllm.ai/en/latest/features/quantization/
- https://docs.vllm.ai/en/latest/features/quantization/llm_compressor/fp8/
- https://docs.pytorch.org/docs/stable/tensor_attributes.html
- https://docs.cloud.google.com/tpu/docs/bfloat16