Self-host Ollama for local LLM inference on Debian 12

You just tried a hosted LLM API and the first thing you noticed is the data policy, then the per-token bill. For a small team or a picky sysadmin, neither is acceptable. That is exactly the situation where you self-host: run Ollama on a Debian 12 VPS, serve a quantized model over an OpenAI-compatible endpoint, and keep every prompt inside your own box. This guide walks you through the install, the systemd service, a safe reverse proxy with Nginx, and the memory math that decides whether your VPS can actually run the model you want.

Prerequisites

• A Debian 12 VPS with at least 4GB RAM. 8GB is comfortable for a 7B parameter model, 16GB lets you run a 13B model without swapping badly.
• Root access or a user with full sudo privileges.
• A domain name pointing at the server, if you want HTTPS through Nginx. You can skip it and use a bare IP with a self-signed cert, but a domain makes Certbot trivial.
• Basic comfort with SSH and systemd. You will edit files under /etc/systemd/system and reload the daemon.

Why run a local LLM on a VPS at all

Self-hosting Ollama is not about beating GPT-4 on a benchmark. It is about control. You decide which model runs, you decide when it updates, and nothing leaves the server. For internal tooling that handles customer data, legal text, or code you cannot send to a third party, that is the whole point. A Linux VPS with full root access lets you pin an exact model version and roll back if a new release changes behavior.

The trade-off is real: a 7B model with 4-bit quantization needs roughly 4 to 5GB of RAM just for weights and context. A 13B model needs about 8 to 10GB. You cannot cheat the math, so the VPS spec decides the model family you can run. That is why the “how much RAM” question comes before the install, not after.

在 Debian VPS 上自托管 Ollama，数据完全留在自己的服务器内。

Self-hosting Ollama on a Debian VPS keeps all data inside your own server.

Step 1 - Installing Ollama on Debian 12

Ollama ships an install script that detects the OS and sets up the binary plus a systemd unit. Run it as root or with sudo:

curl -fsSL https://ollama.com/install.sh | sh

The script places the binary in /usr/local/bin/ollama, creates the ollama user, and installs a systemd service named ollama.service. It listens on 127.0.0.1:11434 by default, which is exactly what you want: the model server should not sit exposed on a public interface.

Verify the service is alive:

systemctl status ollama
# Expected: active (running)

Then pull a small model to confirm the runtime works end to end. Llama 3.2 3B is a good first test because it fits in 4GB:

ollama pull llama3.2:3b
ollama run llama3.2:3b "Say hello in one short sentence"

The first run downloads the model, which can take a few minutes depending on your connection. The second command actually runs inference and prints a reply.

Step 2 - Picking the right model for your RAM

This is where most people get burned. They install Ollama, pull a 70B model, and watch the VPS thrash swap until the OOM killer starts shooting processes. The rule is simple: the model needs to fit in memory with room left for the OS and the context window.

Ollama prints a warning when the model does not fit and asks you to reduce the context length. For a production setup you want to set the context explicitly in a Modelfile or at run time:

ollama run llama3.2:3b --num-ctx 2048

A smaller context window cuts memory usage and speeds up time-to-first-token. For most internal tools, 2048 tokens is plenty. A 16GB VPS is the sweet spot for serious work, and it still costs far less per month than a mid-tier hosted API subscription.

Step 3 - Exposing Ollama safely through Nginx

Ollama listens on localhost, which is fine for a single-user setup but useless if you want to call it from other machines or give your team an endpoint. The safe pattern is a reverse proxy with TLS in front, and authentication at the proxy layer because Ollama has no built-in auth.

Install Nginx and Certbot on Ubuntu 24.04 or Debian 12, whichever your VPS runs:

apt update
apt install nginx python3-certbot-nginx

Create a site config for the proxy:

nano /etc/nginx/sites-available/ollama

server {
    listen 80;
    server_name llm.example.com;

    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Enable the site and generate the certificate:

ln -s /etc/nginx/sites-available/ollama /etc/nginx/sites-enabled/
nginx -t
systemctl reload nginx
certbot --nginx -d llm.example.com

Certbot rewrites the config to enforce HTTPS and sets up auto-renewal. Verify the endpoint answers over the network:

curl https://llm.example.com/api/tags
# Expected: {"models":[...]}

If you see the model list, the proxy works. The endpoint is now public on the internet, so the next step is not optional.

Step 4 - Adding authentication with basic auth

An open API on a public IP will be probed within hours. Ollama does not ship auth, so you put it in Nginx. htpasswd is the fastest path:

apt install apache2-utils
htpasswd -c /etc/nginx/.htpasswd llmuser

Then add two lines inside the location / block in the Nginx config:

auth_basic "Ollama API";
auth_basic_user_file /etc/nginx/.htpasswd;

Reload Nginx and test with credentials:

nginx -t && systemctl reload nginx
curl -u llmuser:yourpassword https://llm.example.com/api/tags

The same credentials work for any OpenAI-compatible client. Set them as the API key in your tooling and you are done. For stronger isolation, put the reverse proxy and Ollama inside a Docker network instead, but for a single-node setup basic auth over TLS is pragmatic and easy to audit.

Step 5 - Managing Ollama as a systemd service

The install script sets a sane default, but you will want to tune it. The main knobs are the bind address, the keep-alive timeout, and the max loaded models. Edit the override file rather than the unit itself:

systemctl edit ollama

[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_KEEP_ALIVE=5m"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_NUM_PARALLEL=1"

Reload and restart:

systemctl daemon-reload
systemctl restart ollama
systemctl status ollama

Explanation of each variable:

• OLLAMA_HOST pins the listener to localhost. The proxy reaches it there, and nothing else can.
• OLLAMA_KEEP_ALIVE unloads the model after 5 minutes of idle, freeing RAM for other work on the box.
• OLLAMA_MAX_LOADED_MODELS=1 prevents two big models from swapping each other out.
• OLLAMA_NUM_PARALLEL=1 serializes requests, which keeps memory predictable.

Check live memory usage to confirm the model fits:

free -h
# Expected: swap should be mostly unused, no OOM kill in dmesg

Troubleshooting

Ollama exits with “CUDA error” or “no GPU” on a VPS. That is expected. A VPS has no GPU, so Ollama falls back to CPU. It works, just slower. If you want GPU inference, you are looking at a dedicated server with a GPU card, not a virtual machine.

The API is unreachable from the internet. First check the proxy:

curl -I https://llm.example.com/api/tags

If that returns 401, auth is working. If it times out, check the firewall. On a Debian 12 host with nftables, allow 80 and 443:

nft add rule inet filter input tcp dport { 80, 443 } accept

Inference is very slow. A 7B model on CPU does a few tokens per second, not hundreds. Reduce the context window (--num-ctx 2048) and close other memory hogs. If the box has a swap file, check whether it is being used during inference with free -h while generating.

FAQ

Related articles

• Self host an LLM API with Ollama on a VPS
• Self-hosting local LLMs on a VPS with Ollama
• Build a self-hosted RAG system with vector DB and LLM
• Run Stable Diffusion image generation on a high-spec VPS

If you need a box for this, a Debian 12 VPS with full root access and a monthly billing plan keeps the setup flexible and the cost predictable. For heavier inference, check the dedicated server options with more RAM and CPU cores.