Scheduling long-running automation jobs on a server you control

You have a script that takes forty minutes to scrape a paginated API, compress the output, and push it to object storage. Or a data pipeline that runs for two hours every night. These are long-running automation jobs, not the kind you run from a terminal and hope the SSH session doesn't drop. The first thing you learn is that cron alone is not enough. It runs a command, but it does not care if that command finishes, or crashes, or silently hangs. If you need an automation server that is actually reliable, you need a stack: cron for scheduling, systemd for supervision, and a supporting cast of service units that keep your jobs alive.

用systemd定时器和进程守护保证长任务失败后自动重启。

systemd timers and process supervision restart a long-running job after it fails.

This guide walks through setting up that stack on a Linux VPS, specifically on Ubuntu 24.04, though the systemd syntax is identical on Debian 12, AlmaLinux 9, and Rocky Linux 9. By the end, you will have a long running automation server that starts jobs on a schedule, restarts them on failure, logs properly, and does not require you to babysit a terminal.

Why a Dedicated Automation Server Beats Running Jobs Ad Hoc

If you are running automation from your local machine, you already know the pain: the VPN drops, the laptop goes to sleep, the job dies at 97% completion. A server that stays on, stays connected, and never goes to sleep fixes that. But the server itself is only half the answer, the other half is the mechanism that launches, monitors, and recovers your jobs.

Cron is the oldest and most widely recognised scheduler on Unix-like systems. It triggers a command at a defined time. But cron has no concept of process state. If a cron job runs a script and that script exits with a non-zero code (or does not exit at all), cron does not retry, does not alert, does not even log what happened by default. That is fine for a five-second maintenance task. For a long running automation server running multi-hour jobs, it is unacceptable.

What you want is a layered approach:
1. Use a systemd timer to trigger the job at a defined time or interval. A timer can specify realtime calendar events, or monotonic offsets from boot, or both. It logs to the journal and reports failure via the same infrastructure as any other service.
2. The timer launches a systemd service unit that runs the actual script. That service unit can have Restart=on-failure, a startup timeout, and dependencies. If the script crashes, systemd sees the non-zero exit and restarts it.
3. Inside the script itself, you implement graceful error handling, and in some cases, a watchdog or heartbeat from the service unit.

This is not new. It is the same pattern used by production deployments for everything from database backups to CI/CD runners. The difference is that most people never explicitly think about it until a job fails silently at 3 AM and nobody notices until morning.

Step 1, Write a Script That Handles Its Own Failure

A long-running job should be idempotent. It should be able to die in the middle, and when it restarts, pick up where it left off or at least not corrupt data. That is a design choice specific to your workflow. But there is one universal rule: your script must exit with a meaningful exit code.

#!/usr/bin/env bash
set -euo pipefail

# Exit codes:
# 0  = success
# 1  = generic error
# 2  = partial failure, retryable
# 3  = critical failure, do not retry without intervention

log() {
  echo "$(date --iso-8601=seconds) $*"
}

main() {
  log "Starting data pipeline"
  # your long-running logic here
  # if something goes wrong, exit 1 or 2
  log "Pipeline completed"
  exit 0
}

main "$@"

Place this script at, say, /opt/automation/pipeline.sh. Make it executable (chmod +x /opt/automation/pipeline.sh). The set -euo pipefail ensures that any failing command inside the script causes an immediate non-zero exit, which is exactly what systemd needs to detect a failure and react.

Step 2, Create the Systemd Service Unit

The service unit tells systemd how to run the script, what to do on failure, and how to handle timeouts. Create /etc/systemd/system/automation-pipeline.service:

[Unit]
Description=Automation data pipeline
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
ExecStart=/opt/automation/pipeline.sh
Restart=on-failure
RestartSec=30
TimeoutStartSec=300
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Key points about this unit:

• Type=exec, modern systemd prefers this over simple for scripts. It waits until the process actually starts executing, not just until it is forked.
• Restart=on-failure, any exit code other than 0 triggers a restart after 30 seconds. You can also use Restart=always, but be careful: that will restart even on intentional shutdowns (like systemctl stop).
• TimeoutStartSec=300, some long-running jobs take a minute to initialise. This prevents systemd from killing them before they even do real work.
• StandardOutput=journal, all output goes to the systemd journal. No log files to manage, no rotation concerns. You read logs with journalctl -u automation-pipeline.

If your job is genuinely continuous (like a message queue consumer that runs forever), add Restart=always and remove the timer, let the service exist as a standalone daemon. But for a scheduled long-running job, you want a timer.

Step 3, Create the Timer Unit

The timer unit does the scheduling. Create /etc/systemd/system/automation-pipeline.timer:

[Unit]
Description=Run automation pipeline daily at 02:00
Requires=automation-pipeline.service

[Timer]
OnCalendar=daily
FixedRandomDelay=300
Persistent=true

[Install]
WantedBy=timers.target

Details:

• OnCalendar=daily, every day at midnight. You can write more specific times: OnCalendar=Mon..Fri 02:00:00 for weekdays at 2 AM, or OnCalendar=*-*-* 00,06,12,18:00:00 for every six hours.
• FixedRandomDelay=300, adds a random delay of up to 300 seconds (5 minutes) so that if multiple timers fire at the same time, they do not all hit the disk at once. This matters more on shared infrastructure where bursts cause latency.
• Persistent=true, if the server was powered off when the timer was supposed to fire, systemd runs it immediately on boot. Critical for a long running automation server that may not stay up 24/7.

Enable and start the timer:

systemctl daemon-reload
systemctl enable automation-pipeline.timer
systemctl start automation-pipeline.timer

Check its status:

systemctl status automation-pipeline.timer
● automation-pipeline.timer - Run automation pipeline daily at 02:00
     Loaded: loaded (/etc/systemd/system/automation-pipeline.timer; enabled)
     Active: active (waiting)
  Trigger: Tue 2026-03-10 02:00:00 ICT; 10h left

You can also list all active timers with systemctl list-timers --all.

Step 4, Add Logging and Alerting

Logs go to the journal by default. But you want to know when a job fails, not just dig through logs after the fact. The simplest approach: on failure, your script can curl a webhook or send an email. The more robust approach: configure systemd's OnFailure directive to trigger a notification service.

Add this line to the [Unit] section of your service unit:

OnFailure=notify-failure@%n.service

Then create a template service /etc/systemd/system/[email protected]:

[Unit]
Description=Failure notification for %i

[Service]
Type=oneshot
ExecStart=/opt/automation/notify.sh %i
StandardOutput=journal
StandardError=journal

The script /opt/automation/notify.sh could send a Telegram message, post to a Slack webhook, or send an email via curl or mailx. This gives you near-instant awareness of a failed job, the only gap is if the server itself is unreachable, in which case a cron-based healthcheck from an external monitoring service (like Uptime Kuma) fills the gap.

Step 5, Handle the Continuous-Process Pattern

Some automation jobs are not batch jobs, they are workers that pull work from a queue and process it continuously. For example, an n8n instance that listens for webhooks and runs workflows in response, or an OpenClaw VPS deployment that runs multi-threaded automation tasks. These processes should never stop. The service unit for that scenario looks different:

[Service]
Type=simple
ExecStart=/opt/automation/worker.py
Restart=always
RestartSec=10
TimeoutStopSec=30
StandardOutput=journal
StandardError=journal

No timer needed because the process runs forever. Restart=always ensures that if the worker crashes (or is killed by OOM killer), systemd brings it back within ten seconds. TimeoutStopSec=30 gives it time to finish the current work item before systemd sends SIGKILL.

This pattern is what runs many of the automation tools you rely on: n8n workers, Redis queue consumers, Node.js long-running processes, Python celery workers. They all benefit from being managed as systemd services rather than as detached nohup or screen processes.

Step 6, When to Use Cron (Yes, Still)

Systemd timers are better in almost every way for managed services. But cron still has one advantage: it is simpler for trivial jobs that run for a few seconds and do not require recovery. If your job is "ping a healthcheck URL every five minutes", cron is fine. If your job is "scrape a site for an hour and send the results", use systemd.

If you do use cron, at least make it report failures. A minimal entry in /etc/crontab:

0 2 * * * root /opt/automation/pipeline.sh || curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/your-uuid/fail

This uses a healthcheck service (like Healthchecks.io or a self-hosted Uptime Kuma instance) to report success, and if the script fails, the || triggers a failure ping. It is a one-liner and costs nothing. But it still has no restart logic: if the job fails, it just reports failure and waits for the next cron trigger.

Step 7, Supervision Beyond Systemd (Optional)

Systemd is built into every modern Linux distribution, which makes it the obvious choice. But some operators prefer a dedicated process supervisor like Supervisor (supervisord) or runit. These tools can be useful if you run multiple workers that share a configuration and need to be started/stopped as a group, or if you want a web UI to see status.

Supervisor, for example, allows you to define a group of processes and start them all with supervisorctl start all. But for most cases, systemd services are simpler to maintain because they follow the same configuration style as every other service on the machine. Mixing systemd for the base OS and Supervisor for your app workers introduces another dependency to audit. Unless you have a specific need (like running workers inside a container without systemd), stick with systemd.

FAQ

Related articles

• Self-hosting n8n for workflow automation on a VPS
• The best AI automation tool in 2026 for real workflows
• Set up a Prometheus Alertmanager pipeline on a VPS
• Using tmux for persistent terminal sessions on a VPS