# A private ChatGPT for your office: Open WebUI + Ollama on your own hardware > OnPrem World · https://onpremworld.com/en/guides/private-chatgpt-open-webui · Updated: 2026-10-14 · CC BY 4.0 **Put Ollama and Open WebUI on one server with a capable GPU, and your team gets a ChatGPT-style assistant in the browser whose prompts and documents never leave your network.** Ollama runs the models on the server's GPU. Open WebUI, started as a single Docker container, adds what an office needs: user accounts, groups and permissions, chat history, and question answering over your own documents. Staff open a web address on the internal network; nothing goes to an outside AI service as long as you use local models and keep Ollama's cloud models and Open WebUI's web search switched off. A server with one 24 GB graphics card runs today's popular 27–35B models for a small team, and you can have the whole setup running in an afternoon. The steps below follow the official Open WebUI (v0.12.0) and Ollama (v0.40.4) documentation as of October 2026. ## What you are building - **Ollama** runs on the server and loads models onto the GPU. It listens on port 11434 and, by default, only accepts connections from the server itself. - **Open WebUI** runs in Docker on the same server, talks to Ollama, and serves the web interface on port 3000. Its data (users, chats, uploaded files) lives in a Docker volume called `open-webui`. - **Staff** use a browser on the office network, or over a VPN when they are away. Nothing is exposed to the internet. ## Step 0: Size the hardware Two things decide the hardware: the model you want, and how many people will use it at the same time. **The model.** Office work such as drafting, summarising and answering questions about documents benefits from the 27–35B class or larger, and document Q&A needs a longer context than casual chat. With a 16K context, a 24 GB card such as the [GeForce RTX 4090](https://onpremworld.com/en/hardware/rtx-4090.md) can run: | Model | Parameters | Best quant that fits | Memory needed | tok/s (est.) | |---|---:|---|---:|---:| | [gpt-oss-20b](https://onpremworld.com/en/models/gpt-oss-20b.md) | 21B (3.6B active) | Q8_0 | 22.7 GiB | 184 | | [Qwen3.6-27B](https://onpremworld.com/en/models/qwen3.6-27b.md) | 27B | Q6_K | 23.2 GiB | 32 | | [Gemma 4 31B IT](https://onpremworld.com/en/models/gemma-4-31b.md) | 30.7B | Q5_K_M | 23.3 GiB | 32 | | [Qwen3.6-35B-A3B](https://onpremworld.com/en/models/qwen3.6-35b-a3b.md) | 35B (3B active) | Q4_K_M | 21.6 GiB | 388 | | [Mistral Small 3.2 24B Instruct (2506)](https://onpremworld.com/en/models/mistral-small-3.2-24b.md) | 24B | Q6_K | 22.3 GiB | 36 | *Estimates from our [methodology](https://onpremworld.com/en/methodology.md), using data as of 2026-10-14. Real speed depends on the runtime, drivers and prompt length.* A 96 GB workstation card such as the [RTX PRO 6000 Blackwell Workstation Edition](https://onpremworld.com/en/hardware/nvidia-rtx-pro-6000-blackwell.md) moves you up to 70B dense models and 100B-plus mixture-of-experts models: | Model | Parameters | Best quant that fits | Memory needed | tok/s (est.) | |---|---:|---|---:|---:| | [Qwen3.6-35B-A3B](https://onpremworld.com/en/models/qwen3.6-35b-a3b.md) | 35B (3B active) | Q8_0 | 37.2 GiB | 394 | | [Gemma 4 31B IT](https://onpremworld.com/en/models/gemma-4-31b.md) | 30.7B | Q8_0 | 33.8 GiB | 38 | | [Llama 3.3 70B Instruct](https://onpremworld.com/en/models/llama-3.3-70b.md) | 70B | Q8_0 | 78.2 GiB | 17 | | [gpt-oss-120b](https://onpremworld.com/en/models/gpt-oss-120b.md) | 117B (5.1B active) | Q6_K | 95.3 GiB | 299 | | [GLM-4.5-Air](https://onpremworld.com/en/models/glm-4.5-air.md) | 106B (12B active) | Q6_K | 88.8 GiB | 127 | | [Qwen3.5-122B-A10B](https://onpremworld.com/en/models/qwen3.5-122b-a10b.md) | 122B (10B active) | Q5_K_M | 85.7 GiB | 176 | *Estimates from our [methodology](https://onpremworld.com/en/methodology.md), using data as of 2026-10-14. Real speed depends on the runtime, drivers and prompt length.* A Mac Studio with lots of unified memory is a quiet alternative for the large MoE models; see our [Mac guide](https://onpremworld.com/en/guides/mac-local-llm.md). For other combinations, use the [calculator](https://onpremworld.com/en/can-i-run.md) or our [GPU buying guide](https://onpremworld.com/en/guides/choosing-a-gpu.md). **Concurrency.** Ollama processes one request per model at a time by default, and queues the rest. Raising `OLLAMA_NUM_PARALLEL` lets it work on several at once, but Ollama's documentation notes that memory scales with the number of parallel requests times the context length. A team of a few people who use the assistant on and off is usually fine with the defaults; if many people use it at the same moment and queues become a problem, a serving engine such as vLLM is built for that (see our [runtime comparison](https://onpremworld.com/en/guides/ollama-vs-lm-studio-vs-llama-cpp-vs-vllm.md)). ## Step 1: Install and configure Ollama On a Linux server, install Ollama with the official script, which also creates a systemd service: ```bash curl -fsSL https://ollama.com/install.sh | sh ``` Then set a few options for an office server. Open the service override: ```bash sudo systemctl edit ollama.service ``` and add under `[Service]`: ```ini [Service] Environment="OLLAMA_NO_CLOUD=1" Environment="OLLAMA_CONTEXT_LENGTH=16384" ``` `OLLAMA_NO_CLOUD=1` disables Ollama's cloud models and web search, so every model runs on your server. Setting the context length explicitly matters in both directions: on GPUs with less than 24 GiB, Ollama's default is only 4K tokens, too short for documents; on cards with 48 GiB or more the default is 256K, which can reserve far more memory than you need. Apply the change and download a model that fits your GPU: ```bash sudo systemctl daemon-reload sudo systemctl restart ollama ollama pull qwen3.6:27b ``` Ollama's API has no authentication, which is why it listens only on `127.0.0.1` by default. Keep it that way: only Open WebUI should talk to it. ## Step 2: Run Open WebUI with Docker Install Docker on the server, then generate a secret key. Open WebUI uses it to sign logins; if it changes when the container is recreated, everyone is logged out, so set it once and keep it somewhere safe: ```bash openssl rand -hex 32 ``` Start Open WebUI with the command from the official quick start, replacing `your-secret-key` with that value: ```bash docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data -e WEBUI_SECRET_KEY=your-secret-key --name open-webui --restart always ghcr.io/open-webui/open-webui:main ``` What the options do: `-p 3000:8080` serves the interface on port 3000; `--add-host` lets the container reach Ollama on the host; `-v open-webui:...` keeps all data in a named volume that survives updates; `--restart always` brings it back after a reboot. The first start takes about a minute. Follow the log until it reports `Application startup complete`: ```bash docker logs -f open-webui ``` **If Open WebUI cannot reach Ollama.** On Linux, a container cannot reach a service bound only to the host's `127.0.0.1`. Open WebUI's troubleshooting page offers two fixes. One is to run the container on the host network, so Ollama can stay private; the interface is then on port 8080: ```bash docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -e WEBUI_SECRET_KEY=your-secret-key --name open-webui --restart always ghcr.io/open-webui/open-webui:main ``` The other is to make Ollama listen on all interfaces with `OLLAMA_HOST=0.0.0.0:11434`. If you do that, block port 11434 from the rest of the network with your firewall, because anyone who can reach it can use your models without logging in. If you prefer Docker Compose, the [quick start](https://docs.openwebui.com/getting-started/quick-start/) has an equivalent `docker-compose.yml`. ## Step 3: Create the admin account and invite your team Open `http://:3000` from a computer on the office network. The first screen asks you to **create the admin account**: the first account created becomes the administrator, so do this immediately, before anyone else can reach the server. (Alternatively, set `WEBUI_ADMIN_EMAIL` and `WEBUI_ADMIN_PASSWORD` before the first start, and the admin is created automatically.) After that, sign-up is switched off. To let colleagues register: 1. Go to your avatar → **Settings** → **Admin** → **Authentication** and turn on **New Sign Ups**. 2. New accounts start as **Pending**; approve them under **Admin Panel** → **Users**. 3. Create groups (for example by department) under **Admin Panel** → **Users** → **Groups**, and set baseline rights for everyone under **Default Permissions**. Permissions add up: a user gets everything granted by any of their groups. 4. To restrict a model or knowledge base, set it to **Private** and grant read or write access to specific groups. For larger organisations, Open WebUI supports single sign-on through OAuth/OIDC (including Google, Microsoft and GitHub) and LDAP. ## Step 4: Let it answer questions about your documents Open WebUI's knowledge bases turn it into a document assistant: 1. Go to **Workspace** → **Knowledge** → **Create**, and upload files. 2. In a chat, type `#` to pick a knowledge base, or attach it permanently to a model under **Workspace** → **Models**. 3. Retrieval settings such as chunk size live under **Settings** → **Admin** → **Documents**. Documents are split and embedded on your server. By default Open WebUI uses a small local embedding model (`sentence-transformers/all-MiniLM-L6-v2`); it can also use an embedding model served by Ollama (`RAG_EMBEDDING_ENGINE=ollama`), which is worth testing if your documents are not in English. Make sure the chat model's context is long enough to read the retrieved passages; Open WebUI's documentation points to the per-model setting under **Settings** → **Admin** → **Models** → **Advanced Params**, and the Ollama context length you set in step 1 applies too. ## Step 5: A checklist for keeping data in-house - **Local models only.** Keep `OLLAMA_NO_CLOUD=1`, and do not add external API connections in Open WebUI unless you mean to. - **Web search off.** It is off by default (`ENABLE_WEB_SEARCH=False`); turning it on sends search queries to an outside provider. - **Offline mode.** Once the embedding model has been downloaded, `OFFLINE_MODE=True` stops update checks and model downloads from Hugging Face. - **Community sharing off.** `ENABLE_COMMUNITY_SHARING` defaults to true and shows options to share to the Open WebUI community; set it to false for an internal tool. - **Decide on admin access to chats.** `ENABLE_ADMIN_CHAT_ACCESS` defaults to true, so administrators can read users' chats. Set it (and `ENABLE_ADMIN_EXPORT`) to false if your policy requires, and tell staff either way. - **Analytics.** The official Docker image already sets `SCARF_NO_ANALYTICS=true`, `DO_NOT_TRACK=true` and `ANONYMIZED_TELEMETRY=false`. - **Network.** Open WebUI's hardening guide says it is designed for private, trusted networks. Do not publish it on the internet; use a VPN for remote staff, and put it behind a reverse proxy with HTTPS (the docs cover Nginx and Caddy). - **Encryption at rest.** For production, the Open WebUI documentation recommends PostgreSQL with encrypted storage, or an encrypted file system if you stay with the default SQLite database. You can pass these settings as `-e NAME=value` options in the `docker run` command; the [environment variable reference](https://docs.openwebui.com/reference/env-configuration/) lists them all. ## Step 6: Back up and update Everything Open WebUI stores is in the `open-webui` volume. Back it up with a throwaway container: ```bash docker run --rm -v open-webui:/data -v $(pwd):/backup alpine tar czf /backup/openwebui-$(date +%Y%m%d).tar.gz /data ``` To update, remove the container (the volume and its data stay), pull the new image, and start it again with the same command as in step 2: ```bash docker rm -f open-webui docker pull ghcr.io/open-webui/open-webui:main ``` Database migrations only go forward, so always back up first. For a shared team instance, the documentation recommends pinning a specific version tag (`ghcr.io/open-webui/open-webui:vX.Y.Z`) and updating deliberately rather than automatically. Ollama updates by re-running its install script. ## Licences: the software and the models - **Ollama** is MIT-licensed. - **Open WebUI** uses the Open WebUI License: BSD-3 plus a branding clause since version 0.6.6 (April 2025). You may deploy it for any number of internal users free of charge as long as the "Open WebUI" name and logo stay intact. Removing or changing the branding is allowed only for deployments with at most 50 users in any rolling 30-day period, with written permission, or under an enterprise licence. - **Models** each have their own licence. Most popular open models allow commercial use, but check the terms of the one you deploy: | Model | Developer | Licence | Commercial use | |---|---|---|---| | [Qwen3.6-27B](https://onpremworld.com/en/models/qwen3.6-27b.md) | Alibaba Qwen | Apache-2.0 | Yes | | [Qwen3.6-35B-A3B](https://onpremworld.com/en/models/qwen3.6-35b-a3b.md) | Alibaba Qwen | Apache-2.0 | Yes | | [Gemma 4 31B IT](https://onpremworld.com/en/models/gemma-4-31b.md) | Google DeepMind | Apache-2.0 | Yes | | [gpt-oss-20b](https://onpremworld.com/en/models/gpt-oss-20b.md) | OpenAI | Apache-2.0 | Yes | | [gpt-oss-120b](https://onpremworld.com/en/models/gpt-oss-120b.md) | OpenAI | Apache-2.0 | Yes | | [Mistral Small 3.2 24B Instruct (2506)](https://onpremworld.com/en/models/mistral-small-3.2-24b.md) | Mistral AI | Apache-2.0 | Yes | | [Llama 3.3 70B Instruct](https://onpremworld.com/en/models/llama-3.3-70b.md) | Meta | Llama 3.3 Community License | Yes | | [GLM-4.5-Air](https://onpremworld.com/en/models/glm-4.5-air.md) | Zhipu AI (Z.ai) | MIT | Yes | ## If you are in Taiwan Interest in on-premises (地端) AI in Taiwan is driven largely by data protection, and a few official documents are useful context. The Executive Yuan's 2023 reference guideline on generative AI for government agencies bars staff from giving confidential or personal information to generative AI, but allows closed, on-premises deployments to be used according to classification level once the system environment's security has been confirmed. The AI Basic Act (人工智慧基本法), promulgated on 14 January 2026, lists privacy protection and data governance, including data minimisation, among its principles. The Personal Data Protection Act continues to apply to any personal data your staff put into the system. Taiwan's own TAIDE models, developed by the National Applied Research Laboratories, are published on Hugging Face under their own licence terms. Our [Chinese edition of this guide](https://onpremworld.com/zh-tw/guides/private-chatgpt-open-webui.md) covers these in more detail. None of this is legal advice: confirm your obligations with your legal or compliance team. ## Frequently asked questions ### Is Open WebUI free for a company? Yes, for internal use by any number of staff, as long as you keep the Open WebUI branding. Since version 0.6.6 (April 2025) its licence is BSD-3 plus a branding clause: removing or changing the Open WebUI name and logo is allowed only for deployments with at most 50 users in any rolling 30-day period, with written permission, or with an enterprise licence. The project notes that this licence is not OSI-approved open source. ### Do prompts and documents leave our network? Not if you run local models and keep the external features off. Ollama runs models on your server; disable its cloud models with OLLAMA_NO_CLOUD=1. In Open WebUI, web search is off by default and should stay off, and offline mode stops update checks and model downloads. The official Docker image already opts out of analytics. ### What hardware do we need for a team? A server with one 24 GB graphics card runs the popular 27–35B models for a small team; by our estimate Qwen3.6-27B needs about 17.5 GiB at Q4_K_M with a 16K context. A 96 GB workstation card or a large-memory Mac Studio runs 70B dense or 100B-plus mixture-of-experts models. Size for concurrency too: each simultaneous request needs its own context memory. ### Can administrators read employees' chats? By default, yes: Open WebUI's ENABLE_ADMIN_CHAT_ACCESS setting defaults to true. If your policy says otherwise, set it to false, along with ENABLE_ADMIN_EXPORT, and tell staff what the policy is. This is a governance decision as much as a technical one. ### Can it answer questions about our own documents? Yes. Create a knowledge base under Workspace, Knowledge, upload files, then reference it in a chat with # or attach it to a model. Open WebUI embeds documents with a local model by default. Raise the model's context length, because Ollama's 4K default on GPUs under 24 GiB limits how much retrieved text the model can read. ### Does running AI on-premises make us compliant with privacy law? It helps, but it is not a guarantee. Keeping data on your own hardware avoids sending it to a third-party AI service, yet the usual rules on collecting, using and securing personal data still apply. This guide is not legal advice; check your obligations with your legal or compliance team. ## Sources - https://docs.openwebui.com/getting-started/quick-start/ - https://docs.openwebui.com/reference/env-configuration/ - https://docs.openwebui.com/features/authentication-access/rbac/ - https://docs.openwebui.com/features/authentication-access/rbac/groups - https://docs.openwebui.com/features/authentication-access/auth/sso/ - https://docs.openwebui.com/features/authentication-access/auth/ldap - https://docs.openwebui.com/features/workspace/knowledge - https://docs.openwebui.com/features/chat-conversations/rag/ - https://docs.openwebui.com/security/chat-data-privacy-and-encryption - https://docs.openwebui.com/security/accepted-risks/first-run-bootstrap-window - https://docs.openwebui.com/getting-started/updating - https://docs.openwebui.com/reference/https/ - https://docs.openwebui.com/getting-started/advanced-topics/hardening - https://docs.openwebui.com/troubleshooting/connection-error - https://docs.openwebui.com/license - https://github.com/open-webui/open-webui/blob/main/LICENSE - https://github.com/open-webui/open-webui/blob/main/Dockerfile - https://github.com/open-webui/open-webui/releases/tag/v0.12.0 - https://docs.ollama.com/linux - https://docs.ollama.com/faq - https://docs.ollama.com/context-length - https://docs.ollama.com/api/authentication - https://law.moj.gov.tw/LawClass/LawAll.aspx?pcode=I0050021 - https://law.moj.gov.tw/LawClass/LawAll.aspx?pcode=H0160093 - https://www.ey.gov.tw/Page/448DE008087A1971/40c1a925-121d-4b6b-8f40-7e9e1a5401f2 - https://huggingface.co/taide