Self-host an LLM API with Ollama on a VPS

You have built an AI application, and the per-seat cost of calling a hosted model API is starting to hurt. Or you are moving data through a third party and your compliance team is asking questions. Either way, the fix is the same: run the model on your own hardware. Ollama makes this practical on a modest VPS. This guide walks through installing Ollama on Ubuntu 24.04, tuning it to fit the RAM you actually have, exposing the API safely, and hardening it before anything else touches it. By the end you will have a working OpenAI-compatible endpoint that your application can call.

Ollama is a runtime for large language models that packages weights, a quantizer, and an HTTP server into a single binary. It exposes an OpenAI-compatible API, which means most existing SDKs and tools work with a one-line base URL change. This is the fastest path to a self-host LLM API on a VPS that you fully control.

Prerequisites

• A VPS running Ubuntu 24.04 LTS, with at least 4 GB of RAM for a functional 7B model. 8 GB is comfortable; 16 GB lets you run larger or multiple models.
• Root access or a sudo user. All commands below assume root.
• A domain name pointing to the server if you plan to expose the API over HTTPS with a reverse proxy. This is strongly recommended.
• An SSH client. Everything is done over the terminal.

If you are still sizing the machine, note that Ollama keeps the model in RAM. A 7B model quantized to Q4 uses about 4.5 GB. A 13B model needs around 8 GB. Choose a plan with headroom above the model size, and remember the OS and runtime take their own ~500 MB. This matters more than vCPU count.

Why run your own LLM backend on a VPS

The recurring cost of a hosted API is not the only reason to self-host. Latency matters: if your application is an internal tool serving users in Vietnam, a model running in a local datacenter with a Vietnam IPv4 responds faster than one that round-trips to a US region. Data residency is another driver. When customer prompts or documents must not leave your infrastructure, a self-host LLM API is the clean answer.

There is a trade-off you need to accept. A single VPS cannot match the throughput of a GPU cluster. You are trading peak performance for predictability and cost. A 2 GB or 4 GB VPS will run small models but slowly. For production workloads, target at least 8 GB RAM. The setup below uses an 8 GB VPS as the reference point.

自托管 Ollama 提供 OpenAI 兼容接口，数据留在你自己的服务器上。

Self-hosted Ollama provides an OpenAI-compatible endpoint and keeps your data on your own server.

Step 1 - Installing Ollama on Ubuntu 24.04

The official installer downloads the binary and sets up a systemd service. You can run it with a single command:

curl -fsSL https://ollama.com/install.sh | sh

The script detects the OS, downloads the current stable binary (Ollama v0.3 as of 2026), creates a dedicated ollama user, and registers the service. Verify that it is running:

systemctl status ollama

Expected output shows active (running). Confirm the API is listening locally:

curl http://127.0.0.1:11434/api/version

You should get a JSON response containing the version number. At this point Ollama listens only on loopback, which is the safe default. Do not change that yet. The hard part comes next.

Step 2 - Pulling a model that fits your RAM

Pick a model whose quantized size leaves room for the OS and the runtime. On an 8 GB VPS, llama3.1:8b is the practical upper bound. On 16 GB you can step up to a 13B model. Pull it with:

ollama pull llama3.1:8b

The download is several gigabytes and takes time depending on your bandwidth. When it finishes, test that it responds:

ollama run llama3.1:8b "Explain what a VPS is in one sentence."

The model streams its answer to your terminal. This command loads the model into memory on first run, which is why the first response is slower than subsequent ones.

Check memory pressure while the model is loaded. In a second SSH session:

free -h

Watch the available column. If it drops below a few hundred MB, the model is too large for the machine and you should pull a smaller quantized variant. Ollama also supports ollama ps to show which models are loaded and their memory footprint.

Key takeaways

• A 7B model quantized to Q4 needs about 4.5 GB of RAM; size your VPS with headroom above the model size.
• Ollama listens on 127.0.0.1:11434 by default. Keep it that way and expose it only through a reverse proxy with HTTPS.
• Set OLLAMA_KEEP_ALIVE to a short duration on a low-RAM VPS so idle models are unloaded instead of holding memory.
• Use the OpenAI-compatible endpoint at /v1/chat/completions, so existing SDKs work with a base URL change.

Tuning Ollama for a low-RAM VPS

Ollama keeps a loaded model in memory for five minutes after the last request by default. On a 4 GB VPS, this means an idle 7B model pins most of your RAM doing nothing. Set a shorter keep-alive and cap how many models stay loaded:

mkdir -p /etc/systemd/system/ollama.service.d
cat > /etc/systemd/system/ollama.service.d/override.conf <<'EOF'
[Service]
Environment="OLLAMA_KEEP_ALIVE=30s"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_NUM_PARALLEL=1"
EOF
systemctl daemon-reload
systemctl restart ollama

These settings trade a slower warm-up for usable memory. The first request after an idle period pays a load penalty of a few seconds. That is the right trade on a 4 GB or 8 GB machine. On a 16 GB VPS you can relax OLLAMA_KEEP_ALIVE to 5m and raise OLLAMA_NUM_PARALLEL to 2 for concurrent requests.

Verify the new behavior:

systemctl show ollama -p Environment
ollama ps

The first command lists the overridden environment variables. The second shows loaded models. After 30 seconds of no traffic, it should report an empty list.

Exposing the API securely with Nginx and HTTPS

Do not open port 11434 to the world. Ollama has no built-in authentication. The correct pattern is a reverse proxy that terminates TLS and forwards to the local service. Install Nginx and a certificate:

apt update
apt install -y nginx python3-certbot-nginx
certbot --nginx -d llm.example.com

Certbot obtains a Let's Encrypt certificate and writes the server block for you. Then align the Nginx config with the local listener. Create /etc/nginx/sites-available/llm with:

server {
    listen 443 ssl;
    server_name llm.example.com;

    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        limit_except GET POST { deny all; }
    }
}

Enable the site and test the config:

ln -s /etc/nginx/sites-available/llm /etc/nginx/sites-enabled/
nginx -t
systemctl reload nginx

The limit_except directive blocks anything that is not a GET or POST, which stops methods like DELETE that expose administrative behavior. Verify the public endpoint responds:

curl https://llm.example.com/api/version

Hardening Ollama for a public-facing backend

An exposed self-host LLM API without any protection is an open relay for anyone who finds it. Anyone could burn your VPS bandwidth and compute sending prompts. Add an API key check in Nginx before it reaches Ollama. Install nginx-extra and use the auth_request module:

apt install -y libnginx-mod-http-auth-request

Generate a strong key and store it in an Nginx map file:

printf "mysecretkey_%s" "$(openssl rand -hex 16)" > /etc/nginx/ollama.key
chmod 640 /etc/nginx/ollama.key
chown root:www-data /etc/nginx/ollama.key

Add an auth check to the location block:

location / {
    auth_request /auth;
    proxy_pass http://127.0.0.1:11434;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
}
location = /auth {
    internal;
    if ($http_authorization != "Bearer $ollama_key") { return 401; }
    return 204;
}

Load the key at the top of the server block with include /etc/nginx/ollama.key; and give it a variable name such as set $ollama_key "the-key-value";. Test and reload Nginx. Your application now sends an Authorization header, and anything without it gets a 401.

On top of TLS and the API key, bind the system firewall to only the needed ports:

ufw allow OpenSSH
ufw allow 'Nginx Full'
ufw enable

Verify with ufw status verbose. Port 11434 stays closed to the outside, reachable only through Nginx on the loopback interface.

Calling the self-host LLM API from your application

Ollama speaks the OpenAI protocol at /v1/chat/completions. Any client that targets OpenAI works by swapping the base URL. With the OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(
    base_url="https://llm.example.com/v1",
    api_key="mysecretkey_yourgeneratedvalue",
)
resp = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Summarize this log file."}],
)
print(resp.choices[0].message.content)

With curl, the same call looks like:

curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer yourkey" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.1:8b","messages":[{"role":"user","content":"Hello"}]}'

This is the whole integration story. Your application code keeps the OpenAI contract, and the model runs on your infrastructure. For a low-noise deployment on a Linux VPS with full root access, this is a solid production pattern for internal tools and prototypes.

Troubleshooting common failures

Out-of-memory kills. If the process dies under load, check journalctl -u ollama -f for OOM entries. The fix is a smaller model or a shorter keep-alive. You can also add swap, but do not rely on it for inference; it will be slow. On a low-RAM VPS this is the most common mistake, sizing the model to the disk instead of the RAM. A benchmark run tells you what the machine can actually sustain.

Slow first token. The model was unloaded by the keep-alive timer. Raising OLLAMA_KEEP_ALIVE keeps it hot at the cost of RAM. This is a tuning choice, not a bug.

409 or 401 from the API. A 401 means the Authorization header is missing or wrong. A 409 usually means OLLAMA_MAX_LOADED_MODELS=1 is blocking a second concurrent model load. Decide whether you want serialized loads or more RAM usage.

Certbot fails to obtain a cert. Port 80 must be reachable from the internet for the HTTP-01 challenge. Check ufw status and make sure Nginx Full is allowed.

How to keep Ollama updated

The installer does not set up automatic updates. Re-run it manually on your maintenance schedule:

curl -fsSL https://ollama.com/install.sh | sh

The script preserves your models and configuration, replacing only the binary. Check the installed version before and after with ollama --version. Model weights are stored under /usr/share/ollama/.ollama/models by default and survive reinstall.

Should you self-host your LLM API at all

Self-hosting an LLM is the right call when you want data residency, predictable cost at scale, or a fixed model version that you control. It is the wrong call when you need frontier-level reasoning on a single VPS. A 7B or 13B model is capable, but it is not GPT-class. Benchmark your actual workload before committing. If the quality bar is met, the economics favor you strongly: one flat monthly VPS bill instead of per-token pricing. For a workload that touches Vietnam, hosting the model on a Vietnam VPS with a dedicated IPv4 keeps the inference path local to your users.

FAQ

Related articles

• Self-hosting local LLMs on a VPS with Ollama
• Best AI chatbot platforms to self-host in 2026
• Self-hosting n8n for workflow automation on a VPS
• Configure swap and optimize memory on a low-RAM VPS