Three services, one VPS: six things the deploy plan left out

This site is three services: the public Next.js site you are reading, a Next.js admin, and a Go API in front of PostgreSQL. They run as containers behind Caddy on one VPS that I manage myself. No Vercel, no managed database.

The plan fits on a napkin. Each repository has one workflow: on a push to main, run the tests, build an image, push it to GHCR, SSH into the server and restart that one service.

push to main
   │
   ▼
 test ──► build image ──► push to GHCR ──► ssh: pull + up -d ──► verify
                                                 │
            ┌────────────────────────────────────┘
            ▼
   Caddy (80/443, TLS)
     ├─ fayzullayev.uz        → web        (Next.js)
     ├─ admin.fayzullayev.uz  → dashboard  (Next.js)
     └─ api.fayzullayev.uz    → api        (Go)  ──► db (PostgreSQL, internal only)

Everything below is what the napkin left out. None of it is exotic, and none of it was in the plan.

1. The lockfile that worked on my machine

The first CI build failed before it compiled anything:

ERR_PNPM_OUTDATED_LOCKFILE  Cannot install with "frozen-lockfile"

I had added the test tooling, and pnpm install on my laptop had quietly updated node_modules without the lockfile ever being committed in that state. Locally nothing complains, because a normal install just fixes the lockfile for you. CI uses --frozen-lockfile, which refuses to.

That flag is the point, not the problem. It is the only thing guaranteeing that the image contains the dependency versions I tested with. The fix was to regenerate and commit the lockfile, and to run pnpm install --frozen-lockfile locally before a push whenever package.json changes.

2. Two files that disagreed about the Go version

go.mod said go 1.26.4. The Dockerfile said FROM golang:1.23-alpine. My laptop has a current Go, so go build worked. The image build could not.

Two changes: the build image became golang:1.26-alpine, and go.mod went from 1.26.4 to 1.26. A patch version in go.mod pins the toolchain more tightly than this project has any reason to.

3. The site that built empty

The first deploy succeeded and the blog had no posts.

The list pages are statically generated, and generateStaticParams() asks the API for the content. In CI there is no API. I had deliberately made that fetch return an empty list instead of failing, so a build never depends on a running backend - which also means a build happily produces a site with nothing on it.

The fix is not to make the build fail. It is to stop treating build time as the moment content is read: the pages revalidate every 60 seconds, so the first request after a deploy fills them from the real API. Before that change the workaround was restarting the container by hand after every deploy.

4. The deploy that was green and did nothing

A run finished green - test, build, deploy - and the site still showed the old text.

The clue was the duration: the deploy job took twelve seconds. Pulling a new Next.js image does not take twelve seconds.

My health check was curl the domain and expect a 200. The old container answers 200 just as well as a new one. The check proved that something was running, which was never in doubt.

So the deploy now compares what is running with what it just pulled:

want=$(docker image inspect "$REGISTRY/portfolio:$TAG" --format '{{.Id}}')
got=$(docker inspect "$(docker compose ps -q web)" --format '{{.Image}}')
[ "$want" = "$got" ] || exit 1

With that in place the next deploy went red and showed the real cause. The image tag is exported for the whole compose file, and web declares depends_on: api. So up -d web tried to bring up the API at the website's commit hash - an image that does not exist - failed, and left the old web container in place. Adding --no-deps fixed it. Without the image check there was nothing pointing at it.

5. Generated code that was never committed

The API uses sqlc: I write SQL, it generates Go. After a migration I regenerated the code locally, everything compiled, and I pushed. CI failed with six errors of the form p.TranslationKey undefined.

The generated package was on my disk and not in the commit. The test job caught it before an image was built, which is exactly what the test job is for, but six confusing compile errors are a bad way to say "you forgot a git add". CI now says it directly:

sqlc generate
git diff --exit-code internal/db || {
  echo "internal/db is stale. Run 'make generate' and commit it."; exit 1; }

6. The token that expired three weeks later

For three weeks everything worked. Then a deploy failed with:

Error response from daemon: error from registry: denied

The server pulled images using a personal access token from a docker login I had run once during setup. The token had an expiry date. Nothing warns you; the site keeps serving the old container, and the only reason I found out the same day is that check from section 4 turning the run red.

A longer-lived token only moves the date. The workflow now passes its own short-lived token to the server and logs in for that one pull:

auth=$(mktemp -d)
echo "$GHCR_TOKEN" | docker --config "$auth" login ghcr.io -u "$GHCR_USER" --password-stdin
docker --config "$auth" pull "$REGISTRY/portfolio:$TAG"
rm -rf "$auth"

A credential that is created for a run and discarded after it cannot go stale between runs.

What I took from it

Four of these six were the same mistake: I trusted a signal that could not tell success from failure. A green build that used a different lockfile. A 200 from the wrong container. A compile on my machine with files CI never saw. A login that worked until a date nobody was watching.

The rule I deploy by now: a deploy is finished when I can prove what is running, not when the pipeline is green. Compare image ids. Fail loudly on drift. Give credentials the lifetime of the job that uses them.

Managed platforms hide every item on this list, and that is a fair thing to pay for. Having met each of them once, I at least know what I would be paying for.

The deploy was only half of it. The other half - backups, a watchdog and an uptime ping that existed in the repository but not on the server - is in the monitoring I wrote and never switched on.