Runtimes & apps Published: Oct 15, 2026

llama.cpp v0.7.0 adds a GPU cache for offloaded MoE experts and moves the server to port 9931

What happened

llama.cpp v0.7.0 adds `--moe-cache-mib`, which keeps recently used mixture-of-experts experts from host memory in an LRU cache on the GPU; the release notes cite up to about 2.2x faster decoding when experts are offloaded, and the cache can be shared across several GPUs. It also adds LiliCorr speculative decoding, MTP for GLM5-Next, and support for new models including Nemotron Super 3.5 VL, Nemotron Bolt 74B A9B, MiniCPM-V 4.7 and EmbeddingGemma-2. The server now listens on port 9931 by default instead of 8080, though Docker images keep 8080 inside the container.

Why it matters locally: If a MoE model is too big for your graphics card and you offload experts to system memory, this cache can speed up generation, but anything that calls llama-server on port 8080 outside Docker needs its address updated.

Three ways to read it

For the market

Large MoE models that spill into system memory are the usual compromise on a single consumer or workstation card, so a faster offload path could make mid-range GPUs plus plenty of RAM a more reasonable alternative to buying a larger card. It also keeps llama.cpp competitive with serving stacks that assume the whole model fits in GPU memory.

In plain words

Some AI models are made of many small specialists, and when they don't all fit on the graphics card, the extras wait in the computer's main memory. The new version keeps the most-used specialists on the graphics card, like keeping your most-used tools on the desk instead of in the drawer.

For practitioners

Set the cache size with `--moe-cache-mib` (or `moe_cache_size` in the C API) and measure decode speed on your own prompts, since the 2.2x figure is an upper bound from the release notes. Before upgrading, update scripts, reverse proxies and health checks that assume port 8080, and note that formatted chat output from the server is now an array of parts rather than a single string.

Sprout's insight

Sprout compares this with other brands and with where real applications are heading. Free for members.

Source

Read the official source: llama.cpp v0.7.0 release notes (GitHub) ↗

Summaries are written by OnPrem World from the official source. We link to the original and never republish it.

More news

All news →