Tukuk-OS Automation: Backups, Watchdogs and Self-Monitoring

From Tukuk-OS Wiki, the free knowledge base.

Tukuk-OS Automation: Backups, Watchdogs and Self-Monitoring
Type	Encyclopedia article
Subject	Tukuk-OS
Author	Fevian Donald
Published	28 October 2026
Revised	28 October 2026
License	Free to read & cite

How Tukuk-OS stays alive without me staring at a screen: scheduled backups to R2, a watchdog that restarts services, and logs that tell what happened.

Contents

1. Scheduled Backups to Cloudflare R2
2. The Watchdog: A Service That Restarts Itself
3. Logs: The System's Memory
4. Testing Recovery: The Practice Most People Skip
5. Notifications Without Noise
6. What Automation Cannot Replace

Self-hosted servers have one truth: they will fail at the most uncomfortable moment. In the early nights Tukuk-OS ran, I woke up checking my phone every few hours. After a few weeks of that I realised the solution was not more human supervision — it was automation.

Scheduled Backups to Cloudflare R2

The most important part of my automation is the backup script. Every few hours the script exports the post database and search index to a gzip file, then uploads it to Cloudflare R2 (cheap object storage).

Three rules I set for backups:

It must be fully automatic. A backup that requires human memory is not a backup; it is a good intention.

It must be tested. I once ran a full restore in a testing environment before trusting the procedure.

It must be cheap. R2 offers very cheap object storage with no egress fees, so the monthly cost stays at a few cents.

Backup failures are never silent: the script writes results to a log, and consecutive failures show up in my daily check.

The Watchdog: A Service That Restarts Itself

Linux systemd has a great feature: Restart=on-failure. If the Tukuk-OS process exits with an error, systemd waits a few seconds and starts it again. My watchdog configuration adds a second layer: a health check that sends GET /health every two minutes.

If two consecutive checks fail, the service is stopped and forcibly restarted. After repeated failures it is marked as failed so I know something needs attention — not just a process that crashed once and recovered on its own.

I deliberately chose a "restart first, investigate later" approach because most of my failures are temporary: a momentary memory spike, a network timeout, or an error that only occurs during startup. Automatic recovery solves 90 percent of problems before I even notice.

Logs: The System's Memory

Every request, error, and notable decision is written to a log. When something happens at three in the morning the log tells me what occurred without guesswork. I use a simple approach: line-based text logs, filterable with grep, and kept on a rotation (old files are trimmed automatically so the disk never fills).

Three things I look for in the daily log: 500 errors, requests that took too long, and attempts blocked by rate limiting. Patterns in those three usually reveal problems long before users notice them.

Testing Recovery: The Practice Most People Skip

The most commonly skipped part of any backup system is testing it. Most people happily produce backup files, nod, and feel safe — without ever trying to restore one. I was that person until I realised I did not actually know whether the file I produced could be read back.

Since then I have made restore testing a monthly routine. The process is simple: take the latest backup copy, load it into a temporary store, run the application against it, and verify the most important data exists and is consistent. The whole process takes about fifteen minutes.

That test has saved me twice. The first time it revealed that my script had been writing an almost-empty file for days because of an unnoticed path change. The second time it revealed that the restore procedure required a command I had already forgotten — and I wrote it into a procedure file before I could forget it again.

The rule I hold now: an untested backup is just hope stored in gzip format. Fifteen minutes a month is a reasonable price for genuine peace of mind.

Notifications Without Noise

I once tried real-time notifications for every event and went deaf within two days. Now I only want to know when: the service has been down for more than a few minutes, backups have failed three times in a row, or disk usage has crossed a threshold. Those three events alone — all of them mean something that genuinely requires action.

What Automation Cannot Replace

Automation restores services and maintains copies, but it cannot judge whether content is good, whether design is clear, or whether a feature should exist. That is human work, and it is why the Tukuk-OS system is designed to minimise routine maintenance — so my time can go into building and writing.

My advice: start with one automatic backup and one health check. Those two things alone will change the server-hosting experience from something nerve-wracking to something dependable.