Build a self-hosted RAG system with vector DB and LLM

You have documents sitting in a folder, and you want an AI to answer questions about them without sending anything to a third party. A retrieval-augmented generation (RAG) system on your own VPS is the answer. You run a vector database for document embeddings, an LLM for answer generation, and a small pipeline that connects the two. Everything stays on hardware you rent, which matters when the documents contain customer data or internal processes. This guide builds a working RAG stack on a Linux VPS with Ubuntu 24.04, using Ollama for inference and Qdrant for vector search. You will have a query endpoint you can call from your own applications.

Prerequisites

• A Linux VPS with at least 8GB RAM. A 16GB plan is comfortable for a 7B model plus Qdrant. If you are unsure about sizing, a Linux VPS from a provider you control works and you can reinstall the OS any time.
• Ubuntu 24.04 installed. Debian 12 also works, the commands differ only in a few package names.
• Root or sudo access on the server.
• A domain name, optional but recommended. You will expose an API, and a proper domain with TLS beats a raw IP.

Why run RAG on your own VPS instead of using a hosted API

A hosted RAG service sounds convenient, but you pay per token, per vector operation, and per gigabyte of storage. Costs climb fast when you index a real document set and then query it repeatedly. Self-hosting changes the economics. You pay one flat monthly fee for the machine, and the marginal cost of an extra query is close to zero.

There is also the data question. With a hosted API, your documents leave your infrastructure. That is a non-starter for legal documents, medical records, or internal knowledge bases. On a self-hosted VPS with a dedicated IPv4, the data never leaves a machine you control. The trade-off is operational: you maintain the model, the vector index, and the pipeline yourself.

Architecture of a self-hosted RAG system

RAG has four moving parts. First, an embedding model turns text chunks into vectors. Second, a vector database stores those vectors and finds the nearest neighbors to a query. Third, an LLM generates the final answer using the retrieved context. Fourth, a pipeline orchestrates the flow: chunk documents, embed them, store them, embed a query, retrieve, and generate.

For this guide, the components are:

Everything runs on one VPS. That keeps the architecture simple, but it means the machine does double duty: CPU and RAM for the LLM, plus disk I/O for the vector index. An NVMe disk matters here. Vector search is random I/O, and spinning disks or network-attached storage will show latency.

Step 1 - Install Ollama for local LLM inference

Ollama packages both the model runtime and a convenient HTTP API. Install it with the official script, then pull the models you need.

curl -fsSL https://ollama.com/install.sh | sh
systemctl enable ollama
systemctl start ollama

Ollama binds to localhost by default on port 11434. That is the safe setting. You do not want the model API exposed to the internet without authentication, so leave it. The Python pipeline will talk to it over localhost.

Pull the embedding model and a generation model. The 3B model runs well on 8GB of RAM. The 7B model needs 16GB to be comfortable.

ollama pull nomic-embed-text
ollama pull llama3.2:3b

Verify the models are ready:

ollama list

Expected output lists both models with their sizes. Test a generation call directly:

curl http://localhost:11434/api/generate -d '{"model":"llama3.2:3b","prompt":"Say hello","stream":false}'

You should get a JSON response with a response field.

Step 2 - Install Qdrant as the vector database

Qdrant is a purpose-built vector database written in Rust. It is fast, and it has a clean HTTP API. Run it with Docker Compose so the data directory persists.

First install Docker if it is not present:

apt update
apt install -y docker.io docker-compose-v2
systemctl enable docker
systemctl start docker

Create a directory for Qdrant and a compose file:

mkdir -p /opt/qdrant
cat > /opt/qdrant/docker-compose.yml <<'EOF'
services:
  qdrant:
    image: qdrant/qdrant:v1.13.4
    container_name: qdrant
    restart: unless-stopped
    ports:
      - "127.0.0.1:6333:6333"
    volumes:
      - ./data:/qdrant/storage
EOF

Bind Qdrant to localhost, not to 0.0.0.0. The vector database has no authentication built in, so it must sit behind the firewall or on a loopback address. The pipeline will connect over localhost.

Start the container:

cd /opt/qdrant
docker compose up -d

Verify Qdrant is up:

curl http://localhost:6333/collections

An empty JSON object {} means the API is responding. There are no collections yet.

将数据留在自己的 VPS 上，用开源模型与向量库构建 RAG 系统。

Build your RAG system with open-source models and a vector database while keeping the data on your own VPS.

Step 3 - Build the indexing pipeline

Now the real work starts. The pipeline has three stages: read documents, split them into chunks, and embed each chunk into Qdrant.

Create a Python virtual environment and install the client libraries:

apt install -y python3-venv python3-pip
mkdir -p /opt/rag
cd /opt/rag
python3 -m venv venv
source venv/bin/activate
pip install qdrant-client openai requests fastapi uvicorn

The openai package works with Ollama because Ollama exposes an OpenAI-compatible endpoint. Point the base URL at localhost.

Write the indexing script. It reads a directory of markdown or text files, splits them, and inserts vectors:

cat > /opt/rag/index.py <<'EOF'
import os, glob
from openai import OpenAI
from qdrant_client import QdrantClient

OLLAMA_URL = "http://localhost:11434/v1"
EMBED_MODEL = "nomic-embed-text"
COLLECTION = "docs"
CHUNK_SIZE = 800
OVERLAP = 100

client = OpenAI(base_url=OLLAMA_URL, api_key="ollama")
qdrant = QdrantClient(url="http://localhost:6333")

qdrant.recreate_collection(
    collection_name=COLLECTION,
    vectors_config={"size": 768, "distance": "Cosine"},
)

def chunk_text(text, size=CHUNK_SIZE, overlap=OVERLAP):
    words = text.split()
    for i in range(0, len(words), size - overlap):
        yield " ".join(words[i:i + size])

files = glob.glob("/opt/rag/docs/*.md") + glob.glob("/opt/rag/docs/*.txt")
idx = 0
for path in files:
    with open(path, "r") as f:
        text = f.read()
    for chunk in chunk_text(text):
        resp = client.embeddings.create(model=EMBED_MODEL, input=chunk)
        vec = resp.data[0].embedding
        qdrant.upsert(collection_name=COLLECTION,
                      points=[{"id": idx, "vector": vec, "payload": {"text": chunk, "source": os.path.basename(path)}}])
        idx += 1
        print(f"Indexed chunk {idx} from {os.path.basename(path)}")

print(f"Done. {idx} chunks indexed.")
EOF

Create a test document folder and add a file:

mkdir -p /opt/rag/docs
cat > /opt/rag/docs/manual.md <<'EOF'
# Deployment manual
The application deploys with Docker Compose. Run docker compose up -d in the project root.
The database migrates automatically on startup. Backups run nightly at 2am via cron.
Secrets are stored in a .env file. Never commit this file to git.
EOF
cd /opt/rag
source venv/bin/activate
python index.py

Verify the collection now contains vectors:

curl http://localhost:6333/collections/docs

The response shows "points_count" with the number of chunks indexed.

Step 4 - Build the query endpoint

Indexing is useless without retrieval. The query pipeline embeds the user question, searches Qdrant for the closest chunks, then feeds them to the LLM along with the question.

Write the FastAPI app:

cat > /opt/rag/app.py <<'EOF'
from fastapi import FastAPI
from pydantic import BaseModel
from openai import OpenAI
from qdrant_client import QdrantClient

OLLAMA_URL = "http://localhost:11434/v1"
EMBED_MODEL = "nomic-embed-text"
GEN_MODEL = "llama3.2:3b"
COLLECTION = "docs"

client = OpenAI(base_url=OLLAMA_URL, api_key="ollama")
qdrant = QdrantClient(url="http://localhost:6333")
app = FastAPI()

class Query(BaseModel):
    question: str

@app.post("/ask")
def ask(q: Query):
    resp = client.embeddings.create(model=EMBED_MODEL, input=q.question)
    vec = resp.data[0].embedding
    hits = qdrant.search(collection_name=COLLECTION, query_vector=vec, limit=3)
    context = "\n\n".join(h.payload["text"] for h in hits)
    prompt = f"Use the following context to answer the question. If the context lacks the answer, say so.\n\nContext:\n{context}\n\nQuestion: {q.question}\n\nAnswer:"
    out = client.chat.completions.create(model=GEN_MODEL, messages=[{"role": "user", "content": prompt}])
    return {"answer": out.choices[0].message.content,
            "sources": [h.payload["source"] for h in hits]}
EOF

Launch the API behind a reverse proxy. Caddy is the fastest way to get TLS automatically:

apt install -y caddy
cat > /etc/caddy/Caddyfile <<'EOF'
rag.example.com {
    reverse_proxy 127.0.0.1:8000
}
EOF
systemctl restart caddy

Start the API with uvicorn. Run it as a systemd service so it survives reboots:

cat > /etc/systemd/system/rag-api.service <<'EOF'
[Unit]
Description=RAG Query API
After=network.target ollama.service docker.service

[Service]
WorkingDirectory=/opt/rag
ExecStart=/opt/rag/venv/bin/uvicorn app:app --host 127.0.0.1 --port 8000
Restart=always
User=root

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now rag-api

Test the whole pipeline:

curl -X POST https://rag.example.com/ask -H "Content-Type: application/json" -d '{"question":"How do I deploy the application?"}'

Expected output is a JSON object with an answer drawn from the manual you indexed, plus the source file name.

Step 5 - Secure the stack

Three services are now running: Ollama on 11434, Qdrant on 6333, and the API on 8000. Only the API should be reachable from outside, and even that needs protection.

First, verify the firewall only exposes ports 80 and 443. Use nftables or ufw, whichever your OS defaults to.

ufw allow 80/tcp
ufw allow 443/tcp
ufw enable
ufw status

Ports 11434 and 6333 are bound to localhost, so they are already unreachable from the internet. Confirm with ss:

ss -tlnp | grep -E ':(11434|6333|8000)'

All three should show 127.0.0.1 as the bound address, never 0.0.0.0.

Second, add authentication to the API. The simplest approach is an API key checked in middleware:

cat > /opt/rag/auth.py <<'EOF'
from fastapi import Header, HTTPException

API_KEY = "change-this-long-random-string"

def require_key(x_api_key: str = Header(...)):
    if x_api_key != API_KEY:
        raise HTTPException(status_code=401, detail="Invalid API key")
EOF

Then add dependencies=[Depends(require_key)] to the /ask route and restart the service.

Third, keep the models and the OS patched. Ollama and Qdrant both update frequently, and vector databases have had CVEs. Set a monthly reminder to check for new versions.

Troubleshooting

Model loads slowly on the first query. Ollama loads the model into memory on demand. The first request after a restart can take 30 seconds or more. Warm it up with a dummy call after boot, or accept the cold start. A larger dedicated server with more RAM holds the model resident and eliminates most of the delay.

Answers are empty or nonsense. Check the retrieval first. Query Qdrant directly with a test vector and see if it returns meaningful payloads.

curl http://localhost:6333/collections/docs -X PUT \
  -d '{"limit":3, "with_payload": true, "vector": [0.1, 0.2, ...]}'

If the search returns nothing useful, the chunking is too aggressive or the document set has too little content. Reduce CHUNK_SIZE and re-index.

API times out under load. A 3B model generates maybe 20 to 40 tokens per second on CPU. Long answers take time. Set a longer timeout in your client, or move to a GPU-backed dedicated server if latency becomes a problem.

journalctl -u rag-api -n 50

Check this log first when the API misbehaves. It shows Python tracebacks and connection errors to Ollama or Qdrant.

How much RAM does a RAG system need?

RAM is the constraint that matters most. The embedding model is small, around 300MB. Qdrant needs RAM proportional to your vector count, roughly a few hundred MB for a few thousand chunks. The LLM is the giant. A 3B model in 4-bit quantization fits in 3GB of RAM. A 7B model needs 6GB or more.

Add the OS, the vector index, and the Python app, and the real floor is 8GB for the 3B setup, 16GB for anything larger. If your document set is big, budget more.

Which model should you pick for retrieval and generation

The embedding model and the generation model are separate choices. For embeddings, nomic-embed-text is a solid default: it handles long documents well and works with Ollama. If you need multilingual retrieval, try snowflake-arctic-embed for English or look for an embedding model trained on your language.

For generation, match the model to your hardware. A 7B model like Qwen 2.5 7B gives noticeably better reasoning than a 3B one, but it demands 16GB. On a Linux VPS with full root access you control every setting, so experiment with a few models and keep the one that gives the most useful answers on your documents.

FAQ

Related articles

• Self-hosting local LLMs on a VPS with Ollama
• Self-host S3-compatible object storage with MinIO on a VPS
• Self-host VPS monitoring with Prometheus and Grafana