The monitoring I wrote and never switched on

In September I wrote the operational half of this site: a nightly database backup, a watchdog that alerts when the ops bot goes quiet, and a morning report on disk and memory. Reviewed, tested, merged.

In October I logged into the server and ran:

$ crontab -l
no crontab for ibrohim

None of it had ever run. For three weeks the site had no backups at all, and nothing had said a word.

Why nothing told me

The bot does watch for missing backups. More than 26 hours without one and it sends a critical alert. But I had made one deliberate exception: if there has never been a backup, it stays quiet. That is a setup state, not an incident, and an alert that fires every day while you are still configuring things teaches you to ignore it.

On its own, that is a reasonable rule. Combined with a setup step that never happened, it meant the one alert built for exactly this situation was switched off by the very condition it should have caught.

The rule I would write now: a quiet setup state needs an expiry date. "Not configured yet" is fine for a day. After a week it is the problem.

A backup on the same disk

The first run worked, and the bot's message said precisely what it was:

💾 Backup
512K · 1s
Faqat shu serverda - tashqi nusxa sozlanmagan.

The last line is Uzbek for "only on this server - no off-site copy configured".

A dump that sits next to the database it copies protects against a bad migration or a careless DELETE. It does not protect against losing the server - which is the case backups exist for.

So every dump is now copied to Cloudflare R2 with rclone straight after it is written, and the morning message says where the copy went.

The bucket has a lock rule: for thirty days nobody can delete or overwrite an object, including the server's own key. I found out the rule was active when cleaning up a test file:

ERROR : r2test.txt: Couldn't delete: ObjectLockedByBucketPolicy

That error is the feature. If the server is ever compromised, whoever has it can read my backups but cannot erase them.

The first restore

A backup you have never restored is a file. So the next step was to restore one - not on my laptop, but on the server, from the off-site copy, into a throwaway database:

F=$(rclone lsf r2:fayzullayev-backups/portfolio | sort | tail -1)
rclone copy "r2:fayzullayev-backups/portfolio/$F" /tmp/
psql -c "CREATE DATABASE portfolio_restore_test;"
gunzip -c "/tmp/$F" | psql -d portfolio_restore_test -v ON_ERROR_STOP=1
psql -d portfolio_restore_test -c \
  "SELECT (SELECT count(*) FROM posts), (SELECT count(*) FROM projects);"

Ten posts, six projects, one user - the same numbers the dashboard shows. That query is now a monthly check, and the bot reminds me on the first of the month.

Writing the real restore procedure turned up one trap worth sharing. The server's .env says TAG=latest, but CI deploys images tagged with the commit hash, so the latest image on the server can be older than what is actually running. Running docker compose up -d after a restore would quietly recreate the containers from that older image - rolling the code back while restoring the data. The procedure uses docker compose start instead, which brings the existing containers back unchanged.

The ping that was never sent

One gap remained: everything above runs on the same machine. If the server dies, the watchdog dies with it.

The bot had code for this from the beginning. Set HEARTBEAT_PING_URL, and every two minutes it pings an external service, which raises the alarm when the pings stop. My notes said this was "one line in .env".

It was not. The compose file never passed that variable into the bot's container. Adding the line to .env would have changed nothing, silently. One more line in docker-compose.prod.yml fixed it, and the URL now points at healthchecks.io - whose alerts arrive through its own Telegram bot and by email, because an alert about my server cannot depend on my server.

What "done" means now

It is the same lesson as the deploy post, from the other side. A green pipeline says the code is correct. It says nothing about whether the code is running.

For operational work, I now count a piece as finished when I have seen it fire once in production: a backup message from the bot, a file in R2, a restore with matching counts, a check turning green on the uptime service. Everything before that is a plan.