Ops Bot

Context

I run this site on one VPS by myself. There is no on-call rota and no monitoring vendor; if the API starts returning errors at two in the morning, the only person who can find out is me, and only if something tells me.

The problem

Sending a Telegram message when something breaks is an afternoon's work. The hard part is the opposite one: sending few enough messages. A failing query at request rate is a hundred alerts a minute, and a channel that behaves like that gets muted on its first bad night - after which the next real outage arrives in silence.

So the design goal was never "report everything". It was: every message is worth reading, and silence means things are fine.

Decisions

An outbox table, not a function call. The API and the bot are separate containers. The API does not talk to Telegram; it inserts a row into a notifications table and moves on. The bot drains the table. If the bot is down the row waits, and if Telegram is slow the API never notices. A message that keeps failing is parked after five attempts so it cannot block the queue behind it.

The bot reads and writes the database directly. It does not go through the API, on purpose: the moment I most need to ask "what is broken?" is the moment the API cannot answer. A tool for fixing an outage must not depend on the thing that is out.

Errors are grouped before they are announced. Each failure is fingerprinted from its route and its message with ids and numbers stripped, so "post 8f3c… not found" and "post a91b… not found" are one group. A group produces at most one message per fifteen minutes, carrying the count of what happened in between. A separate watch catches the other shape of outage - many different errors, each seen once.

Severity decides when, not whether. Only critical messages may arrive at night. Everything else due between 23:00 and 08:00 is held until 09:00. A recovery is deliberately critical too: an outage reported at 02:00 with its recovery held until morning would make the whole night look down.

No flapping. A health verdict is only announced after two consecutive checks agree. One failed request is usually a dropped packet or a container restarting, and a monitor that reports those teaches its owner to ignore it.

The watcher is watched by something that shares nothing with it. When the bot dies, the silence looks exactly like health. So the bot writes a heartbeat row, and a cron job on the host - plain psql and curl, no bot, no outbox - messages Telegram directly when the heartbeat goes stale.

No Docker socket. Giving the bot a view of the host would make it root over the whole stack for the sake of three numbers. Host metrics, deploys and backups are written into the same table by the host and by CI instead.

Architecture

 API ──INSERT──┐
 CI deploy ────┤                        ┌── outbox loop (10 s) ──► Telegram
 host cron ────┼──► notifications ◄─────┤
 bot itself ───┘        (PostgreSQL)    ├── health watch (2 min) ─► health_checks
                                        ├── scheduler, daily and weekly summary
 API errors ──► error_groups ◄──────────┤── certificate watch (12 h)
                                        └── commands: /status /xatolar /posts …
 
 host cron ── reads bot_heartbeat ──► Telegram   (independent of all of the above)

One Go binary, built into the same image as the API and started with a different entrypoint.

What it does day to day

• Announces deploys, published posts, contact-form messages and new submissions.
• Reports API errors, panics, and errors from the browser, grouped and rate-limited.
• Watches the three public domains and the internal API address, and keeps thirty days of history, so /status can answer "how has it been?" and not only "is it up?".
• Warns at 14, 7 and 3 days before a certificate expires - Caddy renews on its own, so that message means renewal is broken.
• Reports each nightly backup and where its copy went, notices when one did not happen, and when the off-site copy failed.
• Lets me draft, edit, schedule and publish posts from the phone.

Running it, not just writing it

Two things I only learned by operating it.

The watchdog was written three weeks before it ran. The backup, watchdog and host-metrics scripts were merged in September. The crontab on the server stayed empty until October, and nothing complained - because I had deliberately made "no backup has ever happened" a setup state rather than an incident, so that the alert would not cry wolf while I was still configuring things. Sensible on its own; combined with a setup step that never happened, it silenced the one alert built for exactly this. A quiet setup state needs an expiry date.

A setting that existed only in the documentation. The external heartbeat - the one alert that still fires when the whole server is gone - was described as "one environment variable". The compose file never passed that variable into the bot's container, so setting it would have changed nothing, without a word.

Both are closed now. Cron runs; every backup lands in a Cloudflare R2 bucket with a 30-day lock, so even a compromised server cannot erase its own history; a restore from that copy has been tested; and healthchecks.io raises the alarm, through its own Telegram bot, if the pings stop.

It is still one server and one person. The aim was never that nothing fails. It is that when something does, I hear about it from something that did not fail with it.