Set up a Prometheus Alertmanager pipeline on a VPS

Why your monitoring stack needs Alertmanager

Prometheus is great at scraping metrics, but raw metrics alone won't wake you up at 3am. Without a proper alerting pipeline, a disk filling up or a service going down becomes a post-mortem discovery instead of a real-time fix. Alertmanager fills that gap: it takes alerts from Prometheus, deduplicates them, groups them by severity or service, and routes them to your team via Telegram, Slack, email, or a custom webhook.

On a VPS running production workloads, whether it's a Linux VPS hosting a web app, a database, or a set of microservices, this pipeline means you catch issues before users do. In this guide, you will install and configure Alertmanager on Ubuntu 24.04, connect it to an existing Prometheus instance, write alert rules, and send notifications to a Telegram bot.

Prerequisites

• A VPS running Ubuntu 24.04 LTS with a non-root sudo user and root access via SSH key-based authentication. If your server is fresh, follow the hardening SSH on a new VPS guide first.
• Prometheus already installed and running (listening on port 9090). If not, install it first via the official binaries or the Prometheus apt repository.
• An outgoing internet connection on the VPS to download packages and reach the Telegram API.
• A Telegram bot token. Create one via @BotFather and get a chat ID (your own or a group ID) where alerts will be sent.

Step 1, Download and install Alertmanager

Alertmanager is a single binary, provided as a prebuilt tarball. Always check the official releases page for the latest version. At the time of this guide, v0.27.0 is current.

cd /tmp
curl -LO https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar -xzf alertmanager-0.27.0.linux-amd64.tar.gz
sudo cp alertmanager-0.27.0.linux-amd64/{alertmanager,amtool} /usr/local/bin/
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager

Create a dedicated system user for security:

sudo useradd --no-create-home --system --shell /usr/sbin/nologin alertmanager
sudo chown -R alertmanager:alertmanager /etc/alertmanager /var/lib/alertmanager

Verify the installation:

alertmanager --version

You should see the version string with build information, for example alertmanager, version 0.27.0 (branch: ...).

Step 2, Write the Alertmanager configuration

The main config file is /etc/alertmanager/alertmanager.yml. The core concepts are inhibit_rules, receivers, and route. In this setup, we use a single route that sends all alerts to a Telegram receiver.

Start with a minimal config that groups alerts by severity and routes them to Telegram:

sudo tee /etc/alertmanager/alertmanager.yml > /dev/null <<EOF
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'telegram-alerts'

receivers:
- name: 'telegram-alerts'
  telegram_configs:
  - bot_token: 'YOUR_TELEGRAM_BOT_TOKEN'
    chat_id: 123456789
    parse_mode: 'HTML'
    message: '{{ range .Alerts }}{{ .Labels.alertname }}, {{ .Annotations.summary }}{{ end }}'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']
EOF

Replace YOUR_TELEGRAM_BOT_TOKEN and 123456789 with your actual bot token and chat ID. The group_by field prevents alert storms: if the same alert fires on multiple instances, Alertmanager groups them into one notification. The inhibit_rule suppresses a warning alert if a critical one already fires for the same host, reducing noise.

Verify the config syntax:

amtool check-config /etc/alertmanager/alertmanager.yml

If it returns no errors, the file is valid.

Step 3, Create a systemd service for Alertmanager

A systemd unit ensures Alertmanager starts on boot and restarts on failure.

sudo tee /etc/systemd/system/alertmanager.service > /dev/null <<EOF
[Unit]
Description=Alertmanager for Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
    --config.file=/etc/alertmanager/alertmanager.yml \
    --storage.path=/var/lib/alertmanager \
    --data.retention=120h
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target
EOF

Reload systemd, start, and enable the service:

sudo systemctl daemon-reload
sudo systemctl enable --now alertmanager

Verify the service is running:

systemctl status alertmanager

It should show active (running). Also check that it listens on its default port:

ss -tlnp | grep :9093

You should see LISTEN on port 9093.

Step 4, Configure the firewall to allow Alertmanager traffic

Alertmanager's web UI is on port 9093. In a production setup, you would proxy it behind nginx or Caddy with authentication and HTTPS. For initial testing from the VPS itself, just allow the local network traffic. If you need to access it remotely, restrict the source IP.

If using UFW:

sudo ufw allow from 127.0.0.1 to any port 9093 proto tcp comment 'Allow Alertmanager local only'
sudo ufw reload

For remote access (not recommended without a reverse proxy), allow your IP only:

sudo ufw allow from YOUR_IP_ADDRESS to any port 9093 proto tcp comment 'Allow Alertmanager external'

Verify the rules:

sudo ufw status verbose | grep 9093

Step 5, Write Prometheus alerting rules

Prometheus evaluates alerting rules defined in a YAML file. The rules use PromQL expressions to determine when an alert fires. Create a rules file for common VPS issues.

sudo mkdir -p /etc/prometheus/rules
sudo tee /etc/prometheus/rules/vps_alerts.yml > /dev/null <<EOF
groups:
- name: vps_node_alerts
  interval: 1m
  rules:
  - alert: HighCPUUsage
    expr: (100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "CPU usage above 80% on {{ $labels.instance }}"
      description: "CPU usage has been above 80% for the last 5 minutes."

  - alert: HighMemoryUsage
    expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Memory usage above 85% on {{ $labels.instance }}"
      description: "Memory usage is {{ $value | humanizePercentage }}."

  - alert: DiskSpaceLow
    expr: (node_filesystem_avail_bytes{mountpoint="/root",fstype!~"tmpfs|overlay|devtmpfs"} / node_filesystem_size_bytes{mountpoint="/root"}) * 100 < 15
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Disk space below 15% on {{ $labels.instance }} ({{ $labels.device }})"
      description: "Available disk space is {{ $value | humanizePercentage }}."

  - alert: NodeExporterDown
    expr: up{job="node"} == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Node exporter is down on {{ $labels.instance }}"
      description: "Prometheus cannot scrape {{ $labels.instance }}."
EOF

Now tell Prometheus to load this rules file. Edit your Prometheus config (usually /etc/prometheus/prometheus.yml). Inside the rule_files section, add:

rule_files:
  - '/etc/prometheus/rules/vps_alerts.yml'

Restart Prometheus to apply:

sudo systemctl restart prometheus

Verify the rules are loaded:

curl http://localhost:9090/api/v1/rules 2>/dev/null | jq .data.groups[0].rules | head -20

You should see the four rules.

Step 6, Configure Prometheus to send alerts to Alertmanager

Prometheus needs an alerting section in its config pointing to Alertmanager. Open /etc/prometheus/prometheus.yml and add this block:

alerting:
  alertmanagers:
  - static_configs:
      - targets:
        - '127.0.0.1:9093'

The full file should now have the sections: global, alerting, rule_files, scrape_configs. Restart Prometheus again:

sudo systemctl restart prometheus

Verify the connection:

curl http://localhost:9093/api/v2/status 2>/dev/null | jq .

You should see "status": "ok". Also check Prometheus targets:

curl http://localhost:9090/api/v1/alertmanagers 2>/dev/null | jq .data.activeAlertmanagers

The list should not be empty.

Step 7, Test the alert pipeline end to end

Force a test alert to validate the whole chain: Prometheus → alert rule → Alertmanager → Telegram. You can send a test alert directly to Alertmanager's API:

curl -X POST -H "Content-Type: application/json" -d '{
  "labels": {
    "alertname": "TestAlert",
    "severity": "critical",
    "instance": "vps01.example.com"
  },
  "annotations": {
    "summary": "This is a test alert from the VPS",
    "description": "Verifying the alert pipeline works end to end."
  }
}' http://localhost:9093/api/v2/alerts

Check Telegram, you should receive a message that looks like: TestAlert, This is a test alert from the VPS.

Alternative test via system: Simulate a disk fill (but be careful) or stop the node_exporter service to trigger the NodeExporterDown alert:

sudo systemctl stop prometheus-node-exporter

Wait 2 minutes and check Telegram. Then restart it:

sudo systemctl start prometheus-node-exporter

A resolution notice will also be sent after the issue clears, thanks to the resolve_timeout in the global config.

Troubleshooting

Problem: Telegram messages never arrive.
Check the Alertmanager logs:

journalctl -u alertmanager -n 50 --no-pager

Look for lines containing telegram or error. Common causes: wrong bot token, chat ID is a private group but the bot is not a member (add the bot to the group), or the VPS cannot reach api.telegram.org.

Problem: Alerts show in Prometheus UI but never fire or reach Alertmanager.
Check that rule_files is correctly referenced and the rule syntax is valid:

promtool check rules /etc/prometheus/rules/vps_alerts.yml

Also verify that the alerting section in prometheus.yml lists the correct target. Restart both services and watch /var/log/syslog for Prometheus errors.

Problem: Alertmanager restarts in a loop.
Stop the service, check the config manually:

sudo systemctl stop alertmanager
sudo /usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --dry-run 2>&1

Fix any syntax errors from the output, then start the service again.

Can I use Alertmanager with a single VPS?

Yes. Alertmanager works perfectly on a single VPS alongside Prometheus. The resource footprint is small, about 30 MB of RAM under normal load.

Do I need a separate VPS for Alertmanager?

No. It can run on the same VPS as Prometheus. For high-availability setups, you would run two Alertmanager instances with the --cluster flag, but a single instance is sufficient for small to medium workloads.

How do I silence a specific alert?

Use the Alertmanager web UI at http://your-vps-ip:9093 (with a reverse proxy for HTTPS). Go to the "Silences" tab, create a new silence with a matcher like alertname="HighCPUUsage".

Can I add more receivers like email or Slack?

Yes. Add a new entry under receivers: for email (email_configs), Slack (slack_configs), or PagerDuty (pagerduty_configs). Route different severities to different receivers by modifying the route tree.

What ports need to be open on the VPS?

Prometheus uses port 9090, Alertmanager uses 9093, and your application's metrics endpoints. For production, expose only port 443 via a reverse proxy. Our reverse proxy with Caddy guide shows how to do this securely.