Virtualization

Self-Hosting Local LLMs on a VPS with Ollama

BM BizMaC Team Jul 15, 2026 7 min read

Running an open-weight language model on infrastructure you control removes the two biggest frustrations of hosted APIs: per-token billing and sending your prompts to a third party. Ollama has become the default way to do this because it wraps model download, quantization, GPU or CPU dispatch, and an HTTP server behind a single binary. This guide covers a realistic setup on a Linux VPS, including the RAM math that decides which models you can actually run, how to expose the API, and how to keep it from being scanned and abused.

What Ollama gives you and where it runs

Ollama is a self-contained runtime built on top of llama.cpp. It manages a local model store, applies quantization, and exposes an HTTP API on port 11434. It runs the same way whether or not a GPU is present: if it detects a supported NVIDIA or AMD card it offloads layers to VRAM, otherwise it executes entirely on CPU. Most VPS plans ship without a GPU, so this article assumes CPU-only inference and treats GPU as the optional case.

The practical consequence: your bottleneck is system RAM and memory bandwidth, not VRAM. A plan with enough RAM and fast NVMe storage for the model weights is enough to serve small and mid-size models for internal tools, batch summarization, or a private chat endpoint.

Installing Ollama

On a fresh Debian or Ubuntu VPS, the official install script sets up the binary and a systemd service:

curl -fsSL https://ollama.com/install.sh | sh

The installer registers a ollama.service unit and starts the background server. Confirm it is listening:

systemctl status ollama
curl http://localhost:11434/api/version

If you prefer isolation, Ollama also ships an official Docker image, which keeps the model store and runtime out of the host filesystem. Either way the API surface and commands are identical.

Choosing a model and sizing RAM

The Ollama library carries hundreds of variants across families like Llama, Qwen, Mistral, Gemma, DeepSeek, and Phi. For general assistant and coding work on a CPU-only box, the sensible tiers in 2026 are:

  • Llama 3.3 for chat, pulled with ollama run llama3.3.
  • Qwen 2.5, which spans 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B, strong on reasoning, math, and multilingual tasks.
  • Mistral 7B (updated to 0.3) and Mistral Small 3 as a compact, capable baseline.

Quantization and the RAM formula

Models are distributed at several quantization levels. Quantization stores each weight in fewer bits, trading a small amount of quality for a large drop in memory footprint. The most common default is 4-bit (Q4, usually Q4_K_M), which is the sweet spot for CPU inference.

A useful rule of thumb: at Q4, a model needs roughly 0.6 GB of RAM per billion parameters, plus a few hundred megabytes for the KV cache and context. That produces these approximate loading requirements:

  • 7B to 8B at Q4: about 4 to 5 GB of RAM for the weights. It will technically load on an 8 GB box, but that leaves almost no headroom for the OS and risks swapping, so treat 16 GB as comfortable for this tier.
  • 13B to 14B at Q4: plan for at least 16 GB of system RAM.
  • 70B to 72B at Q4_K_M: roughly 40 to 45 GB just to load the weights, so a 64 GB machine is the realistic minimum.

On throughput, expect a 7B Q4_K_M model to produce at least 10 tokens per second on a modern CPU with 16 GB of RAM, which is usable for interactive chat. A 70B model on CPU only runs at roughly 1 to 3 tokens per second: fine for background batch jobs, too slow for live conversation. If interactivity matters and you have no GPU, stay in the 7B to 14B range. This is the workload profile that fits a VPS with enough RAM and NVMe headroom without needing dedicated GPU hardware.

Pulling and running models

Download a model by name, then run it. Tags select size and quantization:

# Pull a specific size and quantization
ollama pull qwen2.5:7b

# Start an interactive session (pulls if not present)
ollama run llama3.3

# List what is installed and remove one to free disk
ollama list
ollama rm qwen2.5:7b

To pin an explicit quantization instead of the default tag, request it directly:

ollama pull qwen2.5:7b-q4_K_M

Model weights live under the Ollama store (by default in the service user's home). Because a 7B model is several gigabytes and a 70B model is tens of gigabytes, keep an eye on disk usage and prefer NVMe-backed storage so the initial load into RAM is fast.

Exposing the API

By default Ollama binds to 127.0.0.1:11434 and only accepts local connections. That is the safe default and, for a lot of setups, exactly what you want: your application on the same server talks to localhost.

Call the generate endpoint for a one-off completion, or the chat endpoint for multi-turn context. Add "stream": false to get a single JSON object instead of a token stream:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.3",
  "prompt": "Explain what a reverse proxy does in one sentence.",
  "stream": false
}'

curl http://localhost:11434/api/chat -d '{
  "model": "qwen2.5:7b",
  "messages": [{ "role": "user", "content": "Summarize this log line." }],
  "stream": false
}'

To reach the API from another machine, change the bind address with the OLLAMA_HOST environment variable. Because the installer runs Ollama under systemd, shell exports are ignored; set it in the unit instead:

sudo systemctl edit ollama.service

Add an override:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Binding to 0.0.0.0 makes the server reachable on every interface, which is the point where security stops being optional.

Securing the endpoint

Ollama has no built-in authentication. Any client that can reach the port can run inference and, worse, hit management endpoints such as model deletion. Internet-wide scans have found well over a hundred thousand exposed Ollama servers, so an open port on a public IP will be found. Do not put the raw API on the internet. Use one of these patterns:

  1. Keep it local. If the consumer runs on the same VPS, leave the default 127.0.0.1 bind and never open the port at all.
  2. Reverse proxy with authentication. Keep Ollama on localhost and put Nginx, Caddy, or Traefik in front to terminate TLS and enforce an API key or Basic Auth header before requests reach 11434. When you proxy, expose only the /api/generate and /api/chat endpoints, not the management routes.
  3. Mesh VPN. For access between machines or offices, a mesh VPN such as Tailscale forwards the port over a private network without opening your firewall to the world.

Whichever you choose, also lock the host firewall so port 11434 is never reachable directly from the public internet:

sudo ufw allow 22/tcp
sudo ufw allow 443/tcp
sudo ufw deny 11434/tcp
sudo ufw enable

With this in place, external clients hit the proxy on 443 with credentials, and the model server itself is only ever addressed over loopback.

Wrap-up

A CPU-only VPS with adequate RAM is enough to self-host a useful local model. Size the machine from the quantization math: budget roughly 0.6 GB per billion parameters at Q4, stay in the 7B to 14B range for interactive speed, and reserve headroom for the OS. Install with the official script, pull a model, and keep the default localhost bind unless you have a concrete reason to expose it. When you do expose it, put it behind a reverse proxy with authentication or a private VPN, and firewall the raw port. That combination gives you a predictable, private inference endpoint with no per-token bill and no data leaving your server.