Runtimes & apps Published:
llama.cpp v0.9.0 adds FP8 quantization, Mistral Large 4 and faster multi-user KV cache

What happened
llama.cpp v0.9.0, released on November 4, adds an OCP FP8 E4M3 quantization type and a new flash-attention operation that makes the unified KV cache efficient when several conversations run at once. It supports four new architectures, Clover, Limite, Metis-1.6 and Mistral Large 4, and adds Chatterbox text-to-speech, Nemotron 3.5 Transcribe for offline transcription in 40 languages, and predefined Pocket TTS voices. Gemma 4 E2B and E4B get large prompt-processing speedups, and the bundled ggml v0.29.0 adds early multi-node tensor parallelism over NCCL.
Why it matters locally: Teams sharing one llama.cpp server among several users get a more efficient KV cache, and Mistral Large 4 and Nemotron 3.5 Transcribe can now run on the most common local runtime.
Three ways to read it
For the market
Adding FP8 and early multi-node tensor parallelism likely pushes llama.cpp further into territory that vLLM and SGLang have owned, namely shared servers and very large models. That could matter most to small teams that prefer llama.cpp's single-binary setup over a full serving stack.
In plain words
llama.cpp is the free engine underneath many apps that run AI on your own computer. This update lets it handle several people at once more smoothly, run a few new models, read meeting recordings into text in 40 languages offline, and speak with ready-made voices.
For practitioners
Try the new F8_E4M3 file type against your current Q8_0 builds on your own prompts before switching, since the notes give no quality or speed figures for it. If you serve several sequences with a unified KV cache, re-check throughput, and note that the TTS endpoint now accepts predefined voice names and GET /props lists them. Multi-node tensor parallelism is described as initial support, so keep it to test machines.
Sprout's insight
Sprout compares this with other brands and with where real applications are heading. Free for members.
Sign in free to read it →Subscribe to Sprout's weekly
One email a week: the top stories and a taste of Sprout’s insights. Unsubscribe any time.
Source
Read the official source: llama.cpp v0.9.0 release notes (GitHub) ↗Summaries are written by OnPrem World from the official source. We link to the original and never republish it.
On this site
More news
- NVIDIA and Microsoft bring Nemotron 3.5 and RTX PRO 6000 Blackwell support to Azure Local
- Microsoft publishes Aion 1.0 Instruct, the 2.3B model that becomes Windows' built-in AI by end of January 2027
- RTX Spark compact desktops from ASUS, Dell and MSI go on sale with up to 128 GB of unified memory
- NVIDIA reports $115.8 billion quarter; highlights include a deal to acquire Hugging Face and a 64GB DGX Spark
- Open WebUI v0.13.4 moves personal connections into Tether, sandboxes agents on your own computer and runs local MCP servers