Implement Centralized Log Management with Grafana Loki on Rocky Linux 9

You have twelve servers, each writing logs to its own disk, and the moment something breaks you are SSH-ing into every box hunting for the error. That workflow stops today. On a Rocky Linux 9 VPS we will stand up a centralized log management stack with Grafana Loki, Promtail, and Grafana so every log line from every host lands in one queryable place. Loki is the right tool for this job because it indexes only labels, not the full log content, which keeps memory usage far lower than a full-text engine like Elasticsearch.

Prerequisites

• A Rocky Linux 9 VPS with at least 2 GB of RAM (4 GB recommended, more if you plan to ingest logs from many hosts).
• Root or sudo access to the server.
• Firewalld enabled and active (default on Rocky Linux 9).
• A domain name pointing to the server, if you want Grafana behind HTTPS without a browser warning.
• Basic familiarity with systemd and reading journalctl output.

Why Loki instead of a full-text log engine

Elasticsearch gives you a magnificent search experience and a hefty memory bill to go with it. A 2 GB RAM Linux VPS would choke running Elasticsearch, Kibana, and Logstash together, not to mention the JVM heap tuning that eats your afternoon. Loki takes a different approach: it stores log content compressed in chunks and indexes only the labels you assign. Queries that filter on labels such as host, service, or level run fast, while full-text search is intentionally slower because Loki scans chunks.

For most operations teams this trade-off is the correct one. You care about "show me all errors from nginx on host web-01 in the last hour," not "find every log line in the entire fleet containing the string 'banana'." The latter is a job for a SIEM or a full-text engine, and the former is exactly what Loki excels at.

Loki 只索引标签而非日志内容，所以内存占用远低于 Elasticsearch。

Loki indexes only labels, not the log content itself, so it uses far less memory than Elasticsearch.

Step 1 - Installing Grafana Loki and Promtail

Grafana Labs publishes RPM packages for both Loki and Promtail. We install them from the official repository so updates come through dnf like any other package.

sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://packages.grafana.com/rpm/repo/grafana.repo
sudo dnf install -y loki promtail

The grafana.repo file ships with a GPG key check enabled, so packages are verified before install. After the install finishes, both binaries land in /usr/bin and systemd units are created but not started.

Verify:

rpm -q loki promtail
/usr/bin/loki --version | head -n 2

Expected output shows the installed package versions, something like loki-3.x.x and promtail-3.x.x. If you see a command not found error, the repository did not load, re-run the config-manager step and check for typos.

Step 2 - Configuring Loki for local storage

Loki ships with a configuration file at /etc/loki/config.yml. The default works for a single-binary deployment, but we tighten it for a small self-managed install. Here is a minimal production-oriented config:

sudo cp /etc/loki/config.yml /etc/loki/config.yml.bak
sudo nano /etc/loki/config.yml

Replace the contents with the following:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /var/lib/loki

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /var/lib/loki/chunks

limits_config:
  retention_period: 720h
  max_query_lookback: 720h
  reject_old_samples: true
  reject_old_samples_max_age: 168h

compactor:
  working_directory: /var/lib/loki/compactor
  compactor_ring:
    kvstore:
      store: inmemory
  retention_enabled: true

The important pieces: schema v13 with the tsdb store is the current supported layout in Loki 3.x, retention_period keeps 30 days of logs, and reject_old_samples_max_age stops out-of-order writes from clients whose clocks drift more than a week. If you use a proxy like Nginx in front of Loki, set http_listen_port to bind on localhost only, otherwise leave it on 3100.

sudo systemctl enable --now loki
sudo systemctl status loki

Verify:

curl -s http://localhost:3100/ready

Expected output is ready. If you get a connection refused, check journalctl -u loki -n 50 for config parse errors.

Step 3 - Configuring Promtail to ship logs

Promtail is the agent that reads log files on each server and pushes them to Loki. On this same VPS we configure it to scrape a few common system and application logs.

sudo cp /etc/promtail/config.yml /etc/promtail/config.yml.bak
sudo nano /etc/promtail/config.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
  - job_name: journal
    journal:
      path: /var/log/journal
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__hostname']
        target_label: 'host'

  - job_name: nginx
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          host: localhost
          __path__: /var/log/nginx/*.log

Two scrape jobs here: one reads the systemd journal, which catches SSH logins, sudo events, and any service that logs to journald, and one tail's nginx access and error logs. The positions.yaml file tracks how far Promtail has read each file, so after a restart it resumes where it left off instead of re-shipping everything.

sudo systemctl enable --now promtail
sudo systemctl status promtail

Verify from the Loki side:

curl -s "http://localhost:3100/loki/api/v1/labels" | head -c 500

You should see JSON output listing label names such as host, job, and filename. If the label list is empty, Promtail is not connected, check journalctl -u promtail -n 50 for connection errors.

Step 4 - Installing Grafana and connecting Loki as a data source

Grafana is the query and visualization front end. Install it from the same Grafana repository we added for Loki.

sudo dnf install -y grafana
sudo systemctl enable --now grafana-server
sudo systemctl status grafana-server

Grafana listens on port 3000 by default. Open the firewall for it now, since we will need browser access:

sudo firewall-cmd --permanent --add-port=3000/tcp
sudo firewall-cmd --reload

Open http://your-server-ip:3000 in a browser. The default login is admin / admin, and Grafana forces a password change on first login. Once inside, go to Connections, Data sources, Add data source, pick Loki, and set the URL to http://localhost:3100. Click Save and test, you should get a green "Data source connected and labels found" message.

Verify: On the data source config page, the test result must show a success message. Then go to Explore, select the Loki data source, and run the query {job="systemd-journal"}. Log lines from the journal stream into the panel.

Step 5 - Querying logs with LogQL

LogQL is Loki's query language. It looks like PromQL with a log-flavored twist. The basic building block is a label selector in curly braces, optionally followed by a filter expression. Here are the queries you will use daily:

{job="nginx"} |= "ERROR"
{host="web-01"} | json | level="error"
{job="systemd-journal"} |= "Failed password" |~ "sshd.*invalid user"

The first query returns every nginx log line containing the string ERROR. The second parses each line as JSON and filters on the level field. The third finds SSH brute-force attempts by matching the Failed password pattern in sshd messages.

For rate-based alerts, LogQL also supports metric queries. This one returns the per-minute rate of 5xx errors from nginx:

sum(rate({job="nginx"} |= " 500 "[1m]))

You can graph this in a dashboard panel and attach an alert threshold. Loki does not send alerts itself, Grafana evaluates the query and fires the notification.

Step 6 - Securing Loki behind Nginx with TLS

Binding Loki and Grafana to the public interface without TLS is a bad idea, logs often contain sensitive data. The cleanest approach on a single box is to put Nginx in front of Grafana and enforce HTTPS, while keeping Loki bound to localhost only. We already set Loki to listen on 127.0.0.1:3100, so Promtail on the same host talks to it directly.

If you need remote servers to ship logs to this Loki instance, do not expose port 3100 raw. Set up a TLS reverse proxy, or better, a WireGuard tunnel between the servers and the VPS. Here is the Nginx server block for Grafana over HTTPS using a Let's Encrypt certificate:

server {
    listen 443 ssl http2;
    server_name logs.yourdomain.com;

    ssl_certificate     /etc/letsencrypt/live/logs.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/logs.yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

sudo firewall-cmd --permanent --remove-port=3000/tcp
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

The firewall change blocks direct access to Grafana and leaves only ports 80 and 443 open. Install a certificate with Certbot on a VPS if you have not already. This is also where a VPS with full root access pays off, you control every layer of the stack instead of trusting a managed logging SaaS with your raw logs.

Troubleshooting common Loki and Promtail issues

The two most common failures I see in this stack are Promtail unable to reach Loki, and Loki rejecting samples as out of order.

Promtail cannot connect to Loki:

journalctl -u promtail -n 30

You will see lines like error sending batch, will retry: context deadline exceeded. This almost always means the clients.url in /etc/promtail/config.yml points to a port that is firewalled or a hostname that does not resolve from the Promtail host. Confirm Loki is listening with ss -tlnp | grep 3100 and that the URL is reachable with curl from the same machine running Promtail.

Loki rejects samples:

level=error msg="error processing request" err="out of order sample"

This happens when a client sends a timestamp older than what Loki already accepted for that stream. NTP drift on the shipping server is the usual culprit. Force a resync with sudo chronyc makestep and consider enabling reject_old_samples on the server, which we already did above, so bad samples get dropped instead of poisoning the index.

Grafana shows no data in Explore:

{job="systemd-journal"} |= ""

Run a bare label selector first. If it returns lines, your filter is wrong, if it returns nothing, check that Promtail is reading files. Add a debug level to a scrape config and watch /var/log/promtail/promtail.log after enabling debug logging in the Promtail config.

FAQ

Related articles

• Self-host VPS monitoring with Prometheus and Grafana
• Configure logrotate and manage logs on a VPS
• AI log analysis and anomaly detection on a Linux VPS
• VPS server monitoring tools and strategies for 2026