# Local AI news

> OnPrem World · https://onpremworld.com/en/news · Latest item: 2026-10-13 · CC BY 4.0

New open models, runtime releases and hardware for running AI on your own machine. Each item is a short summary in our own words, why it matters for local use, and a link to the official source.

RSS: https://onpremworld.com/feed.xml · JSON: https://onpremworld.com/api/v1/news

## [NVIDIA RTX Spark laptops with up to 128 GB of unified memory go on sale October 16](https://onpremworld.com/en/news/2026-10-nvidia-rtx-spark-laptops.md)

2026-10-13 · Hardware

NVIDIA said on October 13 that the first RTX Spark laptops launch on October 16. RTX Spark is a single superchip for Windows 11 PCs that pairs a Grace CPU of up to 20 cores with a Blackwell RTX GPU, with up to 128 GB of unified memory and up to 1 petaflop of FP4 AI compute. NVIDIA says it runs models of up to 120B parameters with a 1M-token context, and laptop makers include ASUS, Dell, HP, Lenovo, Microsoft and MSI. The same announcement introduced Nemotron 3.5 Bolt, a 74B mixture-of-experts model with 9B active parameters that fits in 24 GB using 2-bit quantization.

**Why it matters locally:** It puts DGX Spark-class memory in a laptop, but NVIDIA's porting guide lists about 300 GB/s of bandwidth, so expect generation speeds near DGX Spark and Ryzen AI Max systems rather than a desktop RTX card.

Source: [NVIDIA Blog](https://blogs.nvidia.com/blog/local-ai-rtx-spark-laptops-software-updates/)

## [AMD's next Ryzen AI Halo: Ryzen AI Max+ PRO 495 with up to 192 GB of unified memory](https://onpremworld.com/en/news/2026-10-amd-ryzen-ai-halo-max-plus-pro-495.md)

2026-10-12 · Hardware

AMD introduced its next Ryzen AI Halo developer platform on October 12, built on the Ryzen AI Max+ PRO 495 with 16 Zen 5 CPU cores, 40 RDNA 3.5 GPU compute units and up to 192 GB of LPDDR5x unified memory. Up to 160 GB of that memory can be assigned to the GPU, which AMD says is enough for models of 300 billion parameters and more at 4-bit quantization. It comes with a validated ROCm software stack and is sold through Micro Center, with Windows-based versions listed as coming soon.

**Why it matters locally:** That is 64 GB more than the 128 GB maximum of the Ryzen AI Max+ 395 generation, but at about 273 GB/s of bandwidth big dense models will generate slowly; mixture-of-experts models with few active parameters suit it best.

Source: [AMD blog](https://www.amd.com/en/blogs/2026/meet-the-next-amd-ryzen-ai-halo.html)

## [Open WebUI 0.12 adds two-factor sign-in, shared chats you can reply in, and realtime voice](https://onpremworld.com/en/news/2026-10-open-webui-0-12-two-factor-realtime-voice.md)

2026-10-10 · Runtimes & apps

Open WebUI v0.12.0 lets administrators require an authenticator app for every user, with recovery codes and a command-line reset for lost devices. Shared chats and folders can now allow replies, so several people write in one conversation and see each other's messages as they stream. A new Realtime call mode talks through an OpenAI Realtime voice model and hands real questions to the chat's selected model. The release also fixes several Ollama issues; for example, `max_tokens` sent through the API now limits reply length.

**Why it matters locally:** Two-factor sign-in matters if your Open WebUI is reachable from beyond your own machine; note that Realtime voice calls go through OpenAI's cloud, not a local model.

Source: [Open WebUI v0.12.0 release notes (GitHub)](https://github.com/open-webui/open-webui/releases/tag/v0.12.0)

## [Ollama 0.40.2 upgrades older model downloads in the background and keeps backups](https://onpremworld.com/en/news/2026-10-ollama-0-40-2-background-model-upgrades.md)

2026-10-08 · Runtimes & apps

Ollama v0.40.2 upgrades models downloaded with earlier versions the first time you run them, for better performance and compatibility on its llama.cpp engine. To keep downgrades safe it leaves the original copy on disk as a backup for now, and the release notes include a command that removes only those backups. Ollama says a future release will delete them automatically.

**Why it matters locally:** Upgraded models temporarily take up disk space twice, which adds up for a large library; if you delete the backups and later downgrade below 0.40, you will need to pull those models again.

Source: [Ollama v0.40.2 release notes (GitHub)](https://github.com/ollama/ollama/releases/tag/v0.40.2)

## [EmbeddingGemma 2: a 740M multimodal embedding model for local search and RAG](https://onpremworld.com/en/news/2026-10-embeddinggemma-2.md)

2026-10-06 · Models

Google DeepMind released EmbeddingGemma 2 on October 6, an Apache 2.0 embedding model that maps text, code, images, audio and video into one 768-dimensional space. Text-only use needs just the 270M-parameter text part, while the vision and audio encoders load only when needed, for 740M parameters in total. Vectors can be shortened to 512, 256 or 128 dimensions to save storage, and the context window is 8K tokens. Google lists Ollama, LM Studio, llama.cpp, MLX and vLLM among the supported runtimes.

**Why it matters locally:** It is small enough to run beside your chat model on almost any machine, so a fully local RAG setup can search photos, recordings and documents with one model instead of a cloud embedding API.

Source: [Google blog: EmbeddingGemma 2](https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2)

## [llama.cpp v0.6.0 adds GLM-5.3-Flash, a decision-model API and faster Metal kernels](https://onpremworld.com/en/news/2026-10-llama-cpp-v0-6-0.md)

2026-10-05 · Runtimes & apps

llama.cpp published v0.6.0 on October 5, one of its numbered releases alongside the daily builds. It adds support for Z.ai's 320B GLM-5.3-Flash and the Clef decision model, multi-token-prediction speculative decoding for Qwen4Exp models (about 1.5x faster decoding on DGX Spark), and a /v1/systemone server endpoint for decision models. On Apple GPUs it brings a new flash-attention kernel for F16 KV caches and matrix-multiplication kernels that are up to about 3x faster for speculative and batched decoding. The built-in web UI was also reworked around downloading models from Hugging Face.

**Why it matters locally:** Qwen3.8-Flash-Next (a Qwen4Exp model) and GLM-5.3-Flash can now run as GGUF files, and apps that bundle llama.cpp will pick these changes up as they update.

Source: [llama.cpp v0.6.0 release notes (GitHub)](https://github.com/ggml-org/llama.cpp/releases/tag/v0.6.0)

## [vLLM v0.31 adds `vllm preload` for fast restarts and gates per-request multimodal options](https://onpremworld.com/en/news/2026-10-vllm-v0-31-preload.md)

2026-10-05 · Runtimes & apps

vLLM v0.31.0 introduced a `vllm preload` command that runs a daemon keeping already-quantized weights in GPU memory, so a restarted engine maps them instead of reloading from disk. It also brought draft-model speculative decoding to Model Runner V2 and more speed work for DeepSeek-V4.1-Flash. As a security change, per-request `mm_processor_kwargs` and `media_io_kwargs` are now rejected unless the server starts with `--trust-request-mm-kwargs`.

**Why it matters locally:** Restarting a local vLLM server to change settings no longer means waiting for the weights to load again, but clients that send per-request multimodal options will get errors until you opt in.

Source: [vLLM v0.31.0 release notes (GitHub)](https://github.com/vllm-project/vllm/releases/tag/v0.31.0)

## [Ollama 0.35 adds decision models that answer with probabilities instead of text](https://onpremworld.com/en/news/2026-09-ollama-0-35-decision-models.md)

2026-09-28 · Runtimes & apps

Ollama v0.35.0 added a /v1/systemone endpoint for decision models, based on TypeSafe's Jev API. Instead of writing prose, these models pick one of several options, give the probability that a condition is true, or score input against a rubric, returning probabilities with each answer. The first two are Nimble from Bespoke Labs and Tev1 from Together AI, aimed at jobs such as ticket triage, model routing and content classification.

**Why it matters locally:** Classification and routing tasks that used to need a chat model plus brittle output parsing can run on a small local model that returns a structured label and its probabilities.

Source: [Ollama v0.35.0 release notes (GitHub)](https://github.com/ollama/ollama/releases/tag/v0.35.0)

## [Ollama 0.40 runs supported models on MLX by default on Apple Silicon](https://onpremworld.com/en/news/2026-09-ollama-0-40-mlx-default-apple-silicon.md)

2026-09-25 · Runtimes & apps

From v0.40.0, Ollama automatically runs model architectures that its MLX engine supports on Apple Silicon Macs, with no change to `ollama run`. The release names Qwen3.8, Gemma 4, Qwen3.6 and Qwen3.5 among the covered models, and adds MLX versions of its decision models and of the EmbeddingGemma 2 embedding model. Ollama says it will keep testing and enabling more models.

**Why it matters locally:** Mac users get Apple's own machine-learning framework under the hood without new commands, while models outside the MLX list keep running as before; if a model behaves differently after updating, check this switch first.

Source: [Ollama v0.40.0 release notes (GitHub)](https://github.com/ollama/ollama/releases/tag/v0.40.0)

## [Mac mini with M6 and Mac Studio with M5 Ultra go on sale, from US$899 and US$5,499](https://onpremworld.com/en/news/2026-09-mac-mini-m6-mac-studio-m5-ultra-on-sale.md)

2026-09-22 · Hardware

Apple's new Mac mini and Mac Studio went on sale on September 22. The Mac mini starts at US$899 with M6 (configurable up to 32 GB) or US$1,699 with M5 Pro (up to 64 GB), and the Mac Studio starts at US$2,499 with M5 Max (up to 128 GB) or US$5,499 with M5 Ultra. Apple says the 512 GB Mac Studio configuration is coming in late October.

**Why it matters locally:** Starting prices buy the smallest memory tier, and memory decides which models fit, so price the configuration your model actually needs; the hardware pages show what each tier can run.

Source: [Apple Newsroom](https://www.apple.com/newsroom/2026/09/the-new-mac-mini-and-mac-studio-are-available-today/)

## [Open WebUI 0.11.4 shrinks its slim Docker image to about 175 MB](https://onpremworld.com/en/news/2026-09-open-webui-0-11-4-slim-image.md)

2026-09-21 · Runtimes & apps

Open WebUI v0.11.4 cut its slim container image to about 175 MB, roughly 89% smaller than before, by leaving out the bundled local models and the libraries around them; the standard image also lost about 170 MB. The slim build has no built-in embedding or reranking model, so knowledge bases need embeddings from Ollama, OpenAI or Azure OpenAI. It also searches only through PostgreSQL with pgvector and cannot read PDF or Office files unless an external document extractor is configured.

**Why it matters locally:** If you already run Ollama, the slim image with Ollama embeddings is a much lighter way to host Open WebUI; stay on the standard image if you rely on its built-in document and search features.

Source: [Open WebUI v0.11.4 release notes (GitHub)](https://github.com/open-webui/open-webui/releases/tag/v0.11.4)

## [Ollama 0.34.1: MLX safetensors import is stable, GGUF conversion moves to llama.cpp tools](https://onpremworld.com/en/news/2026-09-ollama-0-34-1-mlx-import-gguf-change.md)

2026-09-14 · Runtimes & apps

Ollama v0.34.1 made `ollama create` from MLX safetensors a supported feature rather than an experiment, and improved MLX memory handling on Apple Silicon. The same release stopped converting safetensors into GGUF inside Ollama, so conversion and quantization now have to be done with llama.cpp's own tools first. Listing a large model library through /api/tags also became much faster, dropping from 3.1 s to 294 ms in Ollama's testing.

**Why it matters locally:** Scripts that use `ollama create` to turn safetensors into quantized GGUF files now need a llama.cpp conversion step, while Mac users can import MLX safetensors directly.

Source: [Ollama v0.34.1 release notes (GitHub)](https://github.com/ollama/ollama/releases/tag/v0.34.1)

## [DeepSeek-V4.1-Flash: a 552B multimodal MoE with a quarter of the KV cache](https://onpremworld.com/en/news/2026-09-deepseek-v4-1-flash.md)

2026-09-10 · Models

DeepSeek released V4.1-Flash on September 10, with open weights on Hugging Face under the MIT licence. It is a 552B-parameter mixture-of-experts model that understands images natively and activates about 8B parameters per token while reading the prompt and 16B while generating. A new causal encoder-decoder layout and a second version of DeepSeek's compressed sparse attention cut the KV cache per token to about a quarter of V4-Flash's, with contexts of up to one million tokens.

**Why it matters locally:** Long contexts become cheap, but the 552B backbone alone exceeds 256 GB at 4 bits, so local use means a 512 GB Mac Studio or a multi-GPU server; vLLM added support in v0.30.0 on September 22.

Source: [DeepSeek-V4.1-Flash model card (Hugging Face)](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash)

## [vLLM v0.29 makes Model Runner V2 the default and starts retiring V1](https://onpremworld.com/en/news/2026-09-vllm-v0-29-model-runner-v2-default.md)

2026-09-09 · Runtimes & apps

vLLM v0.29.0 switched every model to Model Runner V2 by default, completing a rollout that began with pooling models. The project now considers V1 deprecated and aims to remove it in v0.32. The release also deprecates starting the server with `python -m vllm.entrypoints.openai.api_server` in favour of `vllm serve`, removes ten deprecated model architectures, and adds Qwen3.8-Flash-Next with BF16, FP8 and NVFP4 weights.

**Why it matters locally:** If your home server still launches vLLM through the old api_server module or depends on a V1-only feature, move to `vllm serve` and test on V2 before v0.32 removes the old runner.

Source: [vLLM v0.29.0 release notes (GitHub)](https://github.com/vllm-project/vllm/releases/tag/v0.29.0)

## [Z.ai releases GLM-5.3-Flash: a 320B multimodal MoE with 18B active, MIT licence](https://onpremworld.com/en/news/2026-08-glm-5-3-flash-open-weights.md)

2026-08-26 · Models

Z.ai released GLM-5.3-Flash on August 26, the first natively multimodal model in its GLM-5 series, with open weights on Hugging Face under the MIT licence. It has 320B parameters in total with 18B active per token and, for the first time in a GLM model, combines sparse and linear attention to cut the cost of long contexts. The model card lists vLLM, SGLang, KTransformers and Unsloth among the ways to run it. The bigger GLM-5.3, built on the GLM-5.2 base and improved through post-training, also has open weights, under a separate GLM-5.3 licence.

**Why it matters locally:** Even at 4 bits per parameter the weights alone take more than 160 GB, so this is a model for 256 GB-class Macs or multi-GPU servers; llama.cpp added support in v0.6.0 on October 5.

Source: [GLM-5.3-Flash model card (Hugging Face)](https://huggingface.co/zai-org/GLM-5.3-Flash)

## [Qwen3.8-Flash-Next previews the Qwen4 architecture: 125B MoE with 6B active](https://onpremworld.com/en/news/2026-08-qwen3-8-flash-next-qwen4-preview.md)

2026-08-26 · Models

Qwen released Qwen3.8-Flash-Next on August 26 as an experimental preview of the architecture that will underpin Qwen4. Its language model has 125B mixture-of-experts parameters with 6B active per token, plus a 51B n-gram embedding table and a 4B multi-token-prediction layer, about 180B parameters in the checkpoint. Attention is split between Gated DeltaNet linear layers and a new Qwen Sparse Attention that picks small blocks of context rather than single tokens, and the native context is 262,144 tokens. The weights use the Qwen Community License, which allows commercial use with conditions.

**Why it matters locally:** With only 6B parameters active per token it can generate quickly, but all ~180B stored parameters still have to fit in memory, far more than one consumer GPU holds; runtime support arrived in Ollama 0.33.1 (MLX engine) and llama.cpp v0.6.0.

Source: [Qwen3.8-Flash-Next model card (Hugging Face)](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)

## [Apple announces M6 and M5 Ultra, with up to 512 GB of unified memory at 1.2 TB/s](https://onpremworld.com/en/news/2026-08-apple-m6-m5-ultra-announced.md)

2026-08-25 · Hardware

Apple announced two chips on August 25: M6 for the new Mac mini and M5 Ultra for the new Mac Studio. M5 Ultra joins two dual-die M5 Max chips into Apple's first quad-die design, with up to 80 GPU cores, up to 512 GB of unified memory and up to 1.2 TB/s of memory bandwidth, 50% more than M3 Ultra. M6 is Apple's first 2 nm chip and supports up to 32 GB of memory at up to 170 GB/s. Apple says M5 Ultra can run language models with hundreds of billions of parameters entirely on the device.

**Why it matters locally:** On Macs, generation speed tracks memory bandwidth, so M5 Ultra keeps M3 Ultra's 512 GB ceiling while our bandwidth-based estimate puts it nearly 50% faster on the same model.

Source: [Apple Newsroom](https://www.apple.com/newsroom/2026/08/apple-introduces-m6-and-m5-ultra-for-a-big-leap-in-performance-and-ai-compute/)

## [Open WebUI 0.11.1 lets you approve each tool call before it runs](https://onpremworld.com/en/news/2026-08-open-webui-0-11-1-tool-approval.md)

2026-08-25 · Runtimes & apps

Open WebUI v0.11.1 lets you switch a conversation from running tools freely to asking first, so a model stops and waits for you to allow or deny each tool call. Administrators have to enable the feature, and your choice is remembered for the conversation. The release also adds a built-in tool that lets a model ask you up to three multiple-choice questions before it continues, and rebuilds streaming so replies send only new text instead of the whole message on every update.

**Why it matters locally:** When a local model can run terminal commands or edit files, approving each call keeps you in control without turning tools off entirely.

Source: [Open WebUI v0.11.1 release notes (GitHub)](https://github.com/open-webui/open-webui/releases/tag/v0.11.1)

## [Ollama 0.32.15 roughly halves time to first token by caching model metadata](https://onpremworld.com/en/news/2026-08-ollama-0-32-15-faster-first-token.md)

2026-08-19 · Runtimes & apps

Ollama v0.32.15 caches resolved model metadata between requests, which cut time to first token from about 995 ms to about 524 ms in Ollama's benchmarks. It also fixes a bug where chat and generate requests could hang after a parser error mid-stream, and handles Qwen 3.8 system messages that appear later in a conversation consistently. First-time desktop users get a new onboarding flow.

**Why it matters locally:** Coding agents and editor plugins that send many short requests feel this most, since half a second saved per call adds up over a long session.

Source: [Ollama v0.32.15 release notes (GitHub)](https://github.com/ollama/ollama/releases/tag/v0.32.15)

## [Qwen3.8-27B released: a dense 27B model that reads images and video, under Apache 2.0](https://onpremworld.com/en/news/2026-08-qwen3-8-27b-released.md)

2026-08-14 · Models

Alibaba's Qwen team released Qwen3.8-27B on August 14, two days after the 2.4-trillion-parameter Qwen3.8 flagship, as the generation's compact dense model. It handles text, images and video, keeps thinking mode on by default and lets you tune reasoning depth per request. Only 16 of its 64 layers use standard attention; the rest are Gated DeltaNet linear-attention layers, and the native context is 262,144 tokens. Ollama added it the same day as qwen3.8:27b.

**Why it matters locally:** Only a quarter of its layers keep a KV cache that grows with the context, so long chats need far less memory than with an older 27B model; the model page lists what each quantization needs.

Source: [Qwen3.8-27B model card (Hugging Face)](https://huggingface.co/Qwen/Qwen3.8-27B)

Summaries are written by OnPrem World from the official source. We link to the original and never republish it.