Runtimes & apps Published:
vLLM v0.32 brings 4-bit KV cache to more GPUs and stops allowing every CORS origin by default
What happened
vLLM v0.32.0, released on October 18 with 814 commits from 363 contributors, extends the NVFP4 KV cache to SM8x, SM90 and SM12x GPUs and adds a 4-bit `ultraquant_4bit` KV cache on ROCm with about twice the density of FP8. Startup gets faster with `vllm download-kernels` for precompiled FlashInfer kernels and `vllm preload --preload-autotune`, and with more than one API server each connection now goes to the least-loaded worker. New models include EmbeddingGemma 2, MiniCPM-V 4.7 and Nemotron 3.5 ASR. The release also fixes three security advisories, restricts the default CORS origins and drops Python 3.10.
Why it matters locally: A 4-bit KV cache means the same graphics card can hold longer conversations or serve more users at once, and the stricter defaults make a self-hosted vLLM server safer out of the box.
Three ways to read it
For the market
Bringing a 4-bit KV cache to older SM8x cards and to AMD's ROCm stack means more of the existing installed base can serve long contexts, which could slow the pressure to buy the newest GPUs just for memory. AMD users gain a feature close to what NVIDIA cards get, which likely helps ROCm look like a practical serving option.
In plain words
When an AI model chats, it keeps a running memo of the conversation in the graphics card's memory, and this update lets vLLM write that memo in a much more compact form on more kinds of cards. It is like switching from full sentences to shorthand, so more conversations fit on the same page.
For practitioners
Test the NVFP4 or `ultraquant_4bit` KV cache against your own quality checks before enabling it, since the release notes report trade-offs elsewhere (GLM-5.x's FP8 KV default costs 7-11% TTFT in the PR's benchmark). Before upgrading, move off Python 3.10, set explicit CORS origins if browsers call the API directly, and re-check prompts near the context limit because `max_tokens` is no longer reserved from the context window.
Sprout's insight
Sprout compares this with other brands and with where real applications are heading. Free for members.
Sign in free to read it →Subscribe to Sprout's weekly
One email a week: the top stories and a taste of Sprout’s insights. Unsubscribe any time.
Source
Read the official source: vLLM v0.32.0 release notes (GitHub) ↗Summaries are written by OnPrem World from the official source. We link to the original and never republish it.
More news
- DeepSeek-V4.1-Flash-1017: a same-size update with better agent scores and shorter answers
- Open WebUI 0.12.3 adds end-to-end encrypted chats and one search box for the whole workspace
- llama.cpp v0.7.0 adds a GPU cache for offloaded MoE experts and moves the server to port 9931
- Mistral releases Timestral, time-series forecasting models from 5M to 209M parameters that run on a CPU
- Ollama 0.40.5 adds NVIDIA Nemotron 3.5 Super and the 24 GB Nemotron 3.5 Bolt