How the estimates work

In one sentence

Whether a model runs depends on whether it fits in memory; how fast it runs depends mostly on memory bandwidth. We compare what the model needs (weights + KV cache + runtime overhead) with the memory the hardware can use, then estimate generation speed from bandwidth. These are estimates, not benchmarks.

Memory needed

Weights = parameters × bits per weight ÷ 8. Bits per weight for quantized formats (e.g. Q4_K_M) are approximate GGUF averages.

KV cache = 2 (K and V) × layers that keep a cache × KV heads × head dimension × 2 bytes (FP16) × context tokens. Layer counts, KV heads and the attention layout come from each model's config.json on Hugging Face.

Runtime overhead = 0.5 GiB + 5% of weights, covering the CUDA/Metal context and compute buffers.

total = params×bits/8 + 2×layers×kv_heads×head_dim×2×context + (0.5 GiB + 5%×weights)

Usable memory

Discrete GPUs: all VRAM (overhead is counted on the model side).

Apple Silicon and unified-memory PCs: about 75% is available to the GPU by default. macOS can raise the limit with sysctl, and mini PCs can change the split in the BIOS, so the real ceiling can be higher.

Generation speed

Generating each token reads the active weights once, so decode speed is mostly bandwidth-bound: tokens/s ≈ bandwidth × efficiency ÷ bytes read per token. For MoE models only the active parameters count, which is why they run much faster than dense models of the same total size.

The efficiency factors are empirical, calibrated against public measurements. We do not estimate prompt-processing (prefill) speed, which is compute-bound.

Known limits

The KV cache follows each model's attention layout. Hybrid linear-attention models (Qwen3.5 onwards) count only their full-attention layers. Sliding-window (Gemma, gpt-oss) and chunked (Llama 4) local layers are capped at the window. MLA models (DeepSeek, Kimi, GLM) use the compressed cache size. DeepSeek-V4's compressed sparse attention is not estimated yet.

Not modelled: splitting across several GPUs, partial offload to system RAM, and KV-cache quantization (Q8/Q4 cache).

Speed varies with the runtime (llama.cpp, Ollama, LM Studio, vLLM, MLX), drivers and batch size. Measured results posted in the forum will be used to calibrate.

Current parameters