Run stable diffusion image generation on a high spec VPS

You have a high-spec VPS with 16 GB of RAM, several cores, and a fast NVMe disk. You want to run Stable Diffusion on it, not rent another cloud GPU. This post walks through the setup that actually works on a Linux VPS in 2026, including the driver situation, the choices between SD 3.5, SDXL, and Flux 2, and how to serve the web UI to yourself without exposing a poisoned port to the internet.

Key takeaways

• Stable Diffusion 3.5 uses the MMDiT architecture and understands long prompts well, but it needs roughly 8 GB of VRAM; SDXL remains the safest choice for 6 GB cards and has the largest ecosystem of LoRAs and ControlNets.
• On a VPS without a GPU, the CPU path is workable for SDXL at 512x512 in about 60 to 90 seconds per image on 8 cores; SD 3.5 and Flux 2 are impractically slow without acceleration.
• Install the NVIDIA driver only if your provider offers GPU instances; otherwise rely on CPU inference and pick the model accordingly.

在无 GPU 的高配 VPS 上，SDXL 是 CPU 推理最稳妥的选择。

On a high-spec VPS without a GPU, SDXL is the most reliable choice for CPU inference.

Prerequisites

• A Linux VPS with at least 8 GB of RAM, ideally 16 GB, and 4 or more vCPUs. Storage should be NVMe, because model files are large and loading them repeatedly from a slow disk hurts.
• Ubuntu 24.04 LTS. The setup below works on Debian 12 as well, with minor package name differences.
• A non-root user with sudo rights.
• Python 3.10 or newer, git, and curl. Ubuntu 24.04 ships Python 3.12, which is fine.
• A domain name if you plan to expose the UI over HTTPS, plus a reverse proxy setup.

Why run image generation on a VPS at all

The obvious answer is that you already pay for a machine that sits idle half the day. Instead of spinning up a separate GPU instance for every experiment, you keep a monthly billing VPS running and generate images when you need them. This makes sense for batch jobs, for testing prompts overnight, or for a small internal tool that produces concept art from a text prompt.

The trade-off is speed. A dedicated GPU instance generates an SDXL image in a few seconds. A VPS with 8 vCPUs and no GPU takes a minute or more per image. Stable Diffusion 3.5 and Flux 2 push that to several minutes per image because their transformer backbones are much heavier. So the real question is not "can it run", it is "will the throughput match your use case". For one-off images or a low-volume internal API, a high-spec VPS is enough. For anything resembling production, rent a GPU box.

Step 1, Prepare the system and install dependencies

Start from a clean Ubuntu 24.04 install. Update the package index and install the build tools that Python packages need when they compile native extensions.

sudo apt update
sudo apt upgrade -y
sudo apt install -y build-essential git curl wget python3-venv python3-pip

Verify Python is version 3.10 or newer.

python3 --version

Expected output is Python 3.12.x on Ubuntu 24.04. If your provider offers GPU instances and you chose one, install the NVIDIA driver and the CUDA toolkit now. On a CPU-only box, skip the driver entirely; the PyTorch CPU build works fine.

sudo apt install -y nvidia-driver-550 nvidia-utils-550

Reboot after installing the driver, then run nvidia-smi to confirm the GPU is visible. If the command is not found, the driver did not load; check dmesg | grep -i nvidia.

Step 2, Decide between SD 3.5, SDXL, and Flux 2

Three models dominate in 2026. Stable Diffusion 3.5, released in October 2024, uses a Multimodal Diffusion Transformer, or MMDiT. It follows prompts much better than SDXL and handles text inside images reasonably well. But it needs around 8 GB of VRAM on a GPU, and on pure CPU it is slow enough to be painful.

SDXL, from 2023, is the workhorse. The ecosystem of fine-tunes, LoRAs, and ControlNets is enormous, and the base model runs in 6 GB of VRAM. More importantly for a CPU-only VPS, SDXL at 512x512 is the only model in this list that produces an image in a tolerable time without acceleration.

Flux 2, from Black Forest Labs in late 2025, is the best-looking of the three, especially for photorealism. But it is the heaviest too. On a CPU-only VPS it is not practical; even on a mid-range GPU it needs quantization to run comfortably.

My honest recommendation: on a CPU-only high-spec VPS, install SDXL and stay there. On a GPU VPS, install SD 3.5 for prompt adherence and keep SDXL for the community models.

Step 3, Install Automatic1111 WebUI

The most battle-tested interface is the Automatic1111 web UI. It handles model downloads, prompt presets, and extensions, and it runs as a local web server. Clone the repository and create a dedicated Python environment.

cd ~
git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
cd stable-diffusion-webui
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

The install pulls PyTorch and a long list of dependencies, so it takes a few minutes. When it finishes, download an SDXL model. The official SDXL base model is on Hugging Face; the command below fetches the fp16 variant, which is the right balance of size and quality.

mkdir -p models/Stable-diffusion
wget -O models/Stable-diffusion/sdxl-base.safetensors \
  https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors

Verify the file size is around 6.9 GB. If it is much smaller, the download failed partway; delete and retry.

ls -lh models/Stable-diffusion/

Step 4, Start the web UI and generate your first image

Launch the web UI bound to localhost. Do not expose it directly to the internet in this state, because it has no authentication of its own.

./webui.sh --listen 127.0.0.1 --port 7860 --xformers

The --xformers flag only helps on an NVIDIA GPU. On a CPU-only box, drop it; it does nothing. Wait for the log line that says the server is running at http://127.0.0.1:7860, then open an SSH tunnel from your laptop.

ssh -L 7860:127.0.0.1:7860 user@your-vps-ip

Now open http://127.0.0.1:7860 locally. Change the model to SDXL, enter a prompt, and generate. The first run compiles some kernels and may take longer; subsequent images hit the steady-state speed.

curl -s "http://127.0.0.1:7860/sdapi/v1/txt2img" \
  -H "Content-Type: application/json" \
  -d '{"prompt":"a red fox in a snowy forest, photorealistic","steps":25}' \
  -o /tmp/out.json

This hits the JSON API directly, which is what you actually want if you plan to script generation. The response contains a base64 image; decode it with a small Python one-liner to confirm everything works.

Step 5, Serve the UI over HTTPS with a reverse proxy

If you need access from a browser without an SSH tunnel, put Nginx in front and add TLS. This is non-negotiable; the WebUI has no login screen, and leaving it on a public port invites anyone to burn your CPU generating images.

sudo apt install -y nginx certbot python3-certbot-nginx

Create a site configuration that proxies port 7860 and sets the Host header so the WebUI behaves itself.

sudo tee /etc/nginx/sites-available/sd < /dev/null

Write the following into that file, replacing sd.example.com with your domain.

server {
    listen 80;
    server_name sd.example.com;

    location / {
        proxy_pass http://127.0.0.1:7860;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_read_timeout 600s;
    }
}

Enable the site and get a certificate from Let's Encrypt.

sudo ln -s /etc/nginx/sites-available/sd /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
sudo certbot --nginx -d sd.example.com

Verify HTTPS is up. The curl output should include the 200 OK status and the title of the WebUI page.

curl -I https://sd.example.com

Add HTTP basic auth in Nginx too, so a guessed subdomain does not give anyone access. Generate a password file with htpasswd and reference it in the server block.

Troubleshooting

Three failures show up constantly. First, out-of-memory kills. The WebUI plus an SDXL model can exceed 8 GB of RAM during image decoding. Check with journalctl -u or dmesg | grep -i oom. Fix: add a swap file of 8 GB, and consider the --medvram flag which trades speed for memory.

sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Second, the --xformers flag crashes on CPU. The error mentions no module named xformers or a CUDA mismatch. Fix: remove the flag; it is GPU-only.

Third, a model fails to load with a "metadata" error. This usually means the download was corrupted. Fix: delete the file and re-download, then verify the byte size against the Hugging Face page.

FAQ

Related articles

• Self-host an LLM API with Ollama on a VPS
• VPS performance benchmarking with fio, sysbench, iperf3
• Build a self-hosted RAG system with a vector DB and LLM
• Reverse proxy with automatic HTTPS using Caddy