OnPrem World地端世界

Runtimes & apps Published:

llama.cpp v0.9.0 adds FP8 quantization, Mistral Large 4 and faster multi-user KV cache

Illustration: llama.cpp v0.9.0 adds FP8 quantization, Mistral Large 4 and faster multi-user KV cache
Illustration

What happened

llama.cpp v0.9.0, released on November 4, adds an OCP FP8 E4M3 quantization type and a new flash-attention operation that makes the unified KV cache efficient when several conversations run at once. It supports four new architectures, Clover, Limite, Metis-1.6 and Mistral Large 4, and adds Chatterbox text-to-speech, Nemotron 3.5 Transcribe for offline transcription in 40 languages, and predefined Pocket TTS voices. Gemma 4 E2B and E4B get large prompt-processing speedups, and the bundled ggml v0.29.0 adds early multi-node tensor parallelism over NCCL.

Why it matters locally: Teams sharing one llama.cpp server among several users get a more efficient KV cache, and Mistral Large 4 and Nemotron 3.5 Transcribe can now run on the most common local runtime.

Three ways to read it

For the market

Adding FP8 and early multi-node tensor parallelism likely pushes llama.cpp further into territory that vLLM and SGLang have owned, namely shared servers and very large models. That could matter most to small teams that prefer llama.cpp's single-binary setup over a full serving stack.

In plain words

llama.cpp is the free engine underneath many apps that run AI on your own computer. This update lets it handle several people at once more smoothly, run a few new models, read meeting recordings into text in 40 languages offline, and speak with ready-made voices.

For practitioners

Try the new F8_E4M3 file type against your current Q8_0 builds on your own prompts before switching, since the notes give no quality or speed figures for it. If you serve several sequences with a unified KV cache, re-check throughput, and note that the TTS endpoint now accepts predefined voice names and GET /props lists them. Multi-node tensor parallelism is described as initial support, so keep it to test machines.

Sprout's insight

Sprout compares this with other brands and with where real applications are heading. Free for members.

Sign in free to read it →

Subscribe to Sprout's weekly

One email a week: the top stories and a taste of Sprout’s insights. Unsubscribe any time.

Source

Read the official source: llama.cpp v0.9.0 release notes (GitHub) ↗

Summaries are written by OnPrem World from the official source. We link to the original and never republish it.

On this site

More news

All news →