Self-host an API speech to text with Whisper on a VPS

You have a podcast workflow, a meeting transcription bot, or a support system that needs speech transcribed, and every call to a cloud STT service costs you per minute and ships your audio to someone else's servers. The fix is a whisper stt instance you run yourself: a small Linux VPS with 4 GB of RAM runs the optimized faster-whisper engine comfortably and serves a private API that your applications call over HTTP. This guide builds exactly that on Debian 12, using medium or small models depending on how fast you need results.

Prerequisites

• A Debian 12 VPS with at least 2 GB of RAM (4 GB recommended for the medium model). A 2 GB instance works fine for small or base. If you need to compare VPS plans, the VNx2 and VNx4 tiers are the starting points here.
• Root access or a user with sudo privileges.
• Python 3.11+ and pip available on the system.
• A domain or subdomain if you want to expose the API over HTTPS with a reverse proxy. Not required for localhost testing.

Why run your own whisper speech-to-text service in 2026

Cloud transcription APIs charge per audio minute and the price adds up fast if you transcribe daily meetings, support calls, or video content. A self-hosted instance pays for itself after a few thousand minutes, and the audio never leaves your server. That matters when you handle client calls, internal strategy sessions, or any recording covered by an NDA.

The practical choice in 2026 is faster-whisper, a reimplementation of OpenAI's Whisper built on CTranslate2. It runs 4 times faster than the reference implementation on the same hardware and uses a fraction of the memory, which makes it the difference between a usable speech-to-text server on 4 GB of RAM and one that swaps itself to death. The model quality matches the original, so you are not trading accuracy for speed.

越南 VPS 提供稳定的本地网络，适合部署私有的语音转文字服务。

A Vietnam VPS gives you a stable local network for running your own private speech-to-text service.

Step 1 - Install Python, pip, and faster-whisper

Start by updating the package index and installing the Python tooling. Debian 12 ships with Python 3.11, which is current enough for faster-whisper.

sudo apt update
sudo apt install -y python3 python3-pip python3-venv ffmpeg
python3 --version

You need ffmpeg because faster-whisper decodes audio through it. The python3-venv package gives you isolated environments so system packages never clash with your STT dependencies.

Now create a dedicated user to run the service. Running it as root is a bad habit, and a service user keeps the attack surface small.

sudo useradd -r -s /usr/sbin/nologin whisper
sudo mkdir -p /opt/whisper-api
sudo chown whisper:whisper /opt/whisper-api

Create a virtual environment and install faster-whisper inside it. This pulls in CTranslate2 and the model downloader.

sudo -u whisper python3 -m venv /opt/whisper-api/venv
sudo -u whisper /opt/whisper-api/venv/bin/pip install --upgrade pip
sudo -u whisper /opt/whisper-api/venv/bin/pip install faster-whisper

Verify the installation works by importing the library and loading a tiny model. The first run downloads the model from Hugging Face, so give it a minute.

sudo -u whisper /opt/whisper-api/venv/bin/python -c "from faster_whisper import WhisperModel; m = WhisperModel('base', device='cpu', compute_type='int8'); print('faster-whisper OK')"

Expected output: faster-whisper OK. If you see a CUDA error, ignore it, you are running on CPU.

Step 2 - Write the transcription API server

The API needs to accept an audio file, transcribe it, and return the text as JSON. Keep it minimal: a single endpoint that takes a file upload and returns the transcript. Below is the full server, written against FastAPI and uvicorn.

sudo -u whisper /opt/whisper-api/venv/bin/pip install fastapi uvicorn python-multipart

Write the application file:

sudo tee /opt/whisper-api/app.py > /dev/null <<'EOF'
import os
import tempfile
from fastapi import FastAPI, UploadFile, File, HTTPException
from faster_whisper import WhisperModel

MODEL_SIZE = os.getenv("WHISPER_MODEL", "small")
model = WhisperModel(MODEL_SIZE, device="cpu", compute_type="int8")

app = FastAPI(title="whisper-stt")

@app.get("/health")
def health():
    return {"status": "ok", "model": MODEL_SIZE}

@app.post("/transcribe")
async def transcribe(file: UploadFile = File(...)):
    if not file.filename:
        raise HTTPException(400, "missing filename")
    suffix = os.path.splitext(file.filename)[1] or ".mp3"
    with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:
        tmp.write(await file.read())
        tmp_path = tmp.name
    try:
        segments, info = model.transcribe(tmp_path)
        text = "".join(seg.text for seg in segments)
        return {"text": text.strip(), "language": info.language}
    except Exception as exc:
        raise HTTPException(500, str(exc))
    finally:
        os.unlink(tmp_path)
EOF

Read what the script does before running it. WHISPER_MODEL is an environment variable, so you switch model size without editing code. The model is loaded once at startup and reused for every request, which is what makes it fast after the first call. The int8 compute type halves memory use on CPU with almost no accuracy loss.

Set the model to small by default. On a rent Linux VPS with 4 GB of RAM you can run medium, but transcription then takes longer per file. Start small, measure, then decide.

Test the server manually before wiring it into systemd.

sudo -u whisper WHISPER_MODEL=small /opt/whisper-api/venv/bin/uvicorn app:app --host 127.0.0.1 --port 8000

In a second terminal, create a short audio file and send it:

ffmpeg -f lavfi -i "sine=frequency=440:duration=2" -ar 16000 test.wav
curl -F "[email protected]" http://127.0.0.1:8000/transcribe

You will not get words back from a pure tone, but you should get a JSON response with a language field, which proves the pipeline works. Stop uvicorn with Ctrl+C.

Step 3 - Run the STT server as a systemd service

A manually started process dies with your SSH session. Run it as a systemd unit so it starts on boot, restarts on crash, and logs to the journal.

sudo tee /etc/systemd/system/whisper-stt.service > /dev/null <<'EOF'
[Unit]
Description=Whisper Speech-to-Text API
After=network.target

[Service]
User=whisper
Group=whisper
WorkingDirectory=/opt/whisper-api
Environment=WHISPER_MODEL=small
ExecStart=/opt/whisper-api/venv/bin/uvicorn app:app --host 127.0.0.1 --port 8000
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

The NoNewPrivileges=true and PrivateTmp=true lines are basic hardening. The service binds to 127.0.0.1 only, so the API is not exposed to the internet yet, only to local processes. That is the correct default.

sudo systemctl daemon-reload
sudo systemctl enable --now whisper-stt.service
sudo systemctl status whisper-stt.service

Expected output shows Active: active (running). Verify the endpoint responds:

curl -s http://127.0.0.1:8000/health

Expected: {"status":"ok","model":"small"}. If the service fails to start, look at the journal:

journalctl -u whisper-stt.service -n 50 --no-pager

Step 4 - Expose the API with Nginx and HTTPS

Localhost is fine for a script on the same machine, but your other servers and clients need a network path. Put Nginx in front as a reverse proxy. It handles TLS termination, request size limits, and keeps uvicorn safely hidden.

sudo apt install -y nginx
sudo tee /etc/nginx/sites-available/whisper-stt > /dev/null <<'EOF'
server {
    listen 80;
    server_name stt.example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        client_max_body_size 200m;
        proxy_read_timeout 300s;
    }
}
EOF
sudo ln -s /etc/nginx/sites-available/whisper-stt /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Two settings matter here. client_max_body_size 200m allows large audio uploads (a 1-hour recording at 128 kbps is about 57 MB). proxy_read_timeout 300s stops Nginx from killing the connection while the model transcribes a long file; CPU transcription is not instant.

Then issue a TLS certificate with Certbot so traffic is encrypted in transit:

sudo apt install -y certbot python3-certbot-nginx
sudo certbot --nginx -d stt.example.com

Verify HTTPS is active:

curl -I https://stt.example.com/health

Expected: HTTP/2 200 with an alt-svc or cert-related header. If you already automate wildcard SSL with acme.sh, point your normal certificate procedure at this subdomain instead.

Step 5 - Test the API end to end with a real recording

The tone test proved the pipeline, but you want proof it transcribes speech. Generate a spoken sentence with a text-to-speech tool locally or download a public sample, then push it through the API.

curl -F "file=@sample_speech.mp3" https://stt.example.com/transcribe

You should receive JSON with the text field containing the spoken words and a language code. That is your working self-hosted speech-to-text API.

Now measure latency for your real workload. Time a request with a 5-minute audio file:

time curl -F "file=@meeting_5min.mp3" https://stt.example.com/transcribe -o transcript.json

On a 4 vCPU VPS with the small model, expect roughly 20 to 40 seconds for a 5-minute file. With medium, double that. If those numbers are too slow, the options are a larger Linux VPS, a smaller model, or splitting audio into chunks and transcribing in parallel.

Troubleshooting common failures

The service starts but returns 500 on every request. Check the audio file. faster-whisper needs ffmpeg to decode, and some formats like certain AAC variants fail silently. Convert to WAV or MP3 first: ffmpeg -i input.m4a -ar 16000 output.wav. Then look at the journal for the real error: journalctl -u whisper-stt.service -n 50.

Out of memory during model load. The medium model needs about 3 GB of RAM with int8. On a 2 GB VPS it will OOM-kill the process. Fix: switch to small via the environment variable, or add swap. Check your swap configuration to see if the instance has any.

HTTP request times out after 60 seconds. Nginx default proxy_read_timeout is 60 seconds, and a long audio file takes longer than that. Set it to 300s as shown in Step 4, and also set request_timeout in whatever client you use. The server does not time out by default, only the proxy does.

FAQ

Related articles

• Self-hosting local LLMs on a VPS with Ollama
• VPS performance benchmarking with fio, sysbench and iperf3
• Configure swap and optimize memory on a low RAM VPS