How smkabir.com stays up: releases, pointers and a failover that is still on paper

This site is static files served by two small nginx pods in a homelab. That is a deliberately boring design, and it makes the interesting question easy to state: what keeps the site up when something breaks? This post is the honest answer, including the part that does not exist yet.

The shape of it#

Diagram source

direction: right

visitor: "Visitor" { shape: person }
cf: "Cloudflare edge"
tunnel: "Tunnel (cloudflared)"
gw: "Gateway (Cilium)"
pods: "Two blog pods" {
  nginx: "nginx"
  sync: "content-sync sidecar"
}
r2: "R2 bucket" { shape: cylinder }
ci: "GitHub Actions"

visitor -> cf: "HTTPS"
cf -> tunnel
tunnel -> gw
gw -> pods.nginx
pods.sync -> r2: "reads LATEST every 60 s"
ci -> r2: "uploads release,\nthen LATEST, then VERIFIED"

Nothing a visitor requests ever touches a database or a CMS. Pages are files. That is why the failure modes are few and the recovery is mostly "serve the previous files".

The content is not in the image#

The container image holds nginx and a small sync program, and no site content at all. The site itself is a release: a folder in an object-storage bucket named after the build, such as releases/c280dba-38211385696/. Two small text files point into it:

• LATEST names the release the pods should serve.
• VERIFIED names the release that was actually seen serving correctly on the real site.

A new post is therefore not a deployment. There is no commit to the cluster repository, no image build and no rollout. A publish uploads a folder and moves a pointer.

How a pod follows the pointer#

Each pod runs a sidecar that does the same small loop every 60 seconds:

1. Read LATEST. If it is empty, unreadable or not a plain id, do nothing and keep serving.
2. If it is already the live release, stop.
3. Download the release next to the live one, never over it.
4. Refuse it unless index.html exists and version.json says the same id as the folder.
5. Turn the release's redirect list into an nginx map using a strict character allow-list, so a bad entry can never inject configuration.
6. Swap the current symlink in one atomic rename, ask nginx to reload, and delete everything but the live release and the previous one.

If any step fails, the previous release keeps serving. A visitor never sees a half-written site, and R2 being down means "keep what you have".

A release, from push to live#

The publish job is one pipeline, and its order is the point:

1. Build and gate. Formatting, type checks, tests, a privacy scan (no GPS in images, no addresses or phone numbers), the build, and checks on the output: no inline scripts, correct language on every page, no drafts anywhere, every link resolves.
2. Upload to releases/<id>/ and re-verify the upload. A release is immutable: the same id with different content is refused, and re-running is a safe no-op.
3. Move LATEST. The pods start syncing within a minute.
4. Wait until the public site reports the new id. This polls the real URL for up to ten minutes.
5. Move VERIFIED, purge Cloudflare's cache and prune releases beyond the last twenty.

If step 4 never succeeds, the run fails loudly, VERIFIED does not move, and the live site is untouched.

Why there are two pointers#

LATEST says what to serve. VERIFIED says what is known to work. They are separate because the thing you want to fall back to must never be a release nobody has looked at. A bad build can become LATEST for a minute; it cannot become VERIFIED, because VERIFIED only moves after the live site has reported that exact id. Any fallback reads VERIFIED, so a fallback is never a gamble.

What happens when something fails#

The failover that is not built yet#

When the tunnel has no healthy connectors, Cloudflare answers with an error page, and its edge cache only helps on points of presence that happen to be warm. The design is a small Worker in front of the site:

Diagram source

direction: right

visitor: "Visitor" { shape: person }
worker: "Failover Worker (planned)"
origin: "Homelab origin"
r2: "R2: VERIFIED release" { shape: cylinder }

visitor -> worker
worker -> origin: "try first, 4 s timeout"
worker -> r2: "only on 5xx, tunnel error or timeout"

• Reads only. Anything other than GET or HEAD goes straight to the origin; the Worker never invents a write.
• The mirror is VERIFIED. It is the same bucket the pods read, so there is no second copy to keep in sync, and it never serves an unverified build.
• Same headers. The security headers come from one source file that renders both the nginx configuration and the Worker, so a mirrored response carries the same policy as an origin one.
• Visible. Mirror responses carry X-Served-From: mirror, and an external uptime probe is meant to alert when it appears.
• What still fails when the homelab is down: publishing and previews. Reading, the resume, search and the feeds keep working, because they are files.
• The drill: scale the deployment to zero through the cluster repository (Argo CD would undo a manual change within seconds), check the key pages and the header, then revert the commit.

It is on paper because the site only went live today and the failover is the next unit of work, not because it is hard. The release layout was chosen so that it is a small Worker and not a second system.

Rolling back#

• Content: a manual workflow takes a build id, checks the release exists and is consistent, and rewrites LATEST and VERIFIED to it. The pods follow within a minute.
• Code (the image): a tag is never moved. A fix is the next patch, and a rollback is reverting the commit that bumped the tag in the cluster repository.

What has been proven, and what has not#

Proven in production in the last day:

• Object storage: uploads, pointer moves, pruning, and pods reading with a read-only token. The token that can write is a different token, held only by CI.
• GitHub Actions: the content pipeline and the image release, including a vulnerability scan that stopped a release for a HIGH finding in a library the image did not need.
• The cluster path: Argo CD sync, two ready pods on different hosts, the gateway route and the tunnel to the apex.
• Tests that exist: 61 image checks (sidecar, nginx under the cluster's restrictions, redirects), 35 release-script checks, an end-to-end run of the real site through nginx, and unit tests.

Not proven: the failover Worker (it does not exist), a real power cut, and a sustained outage of the object store.

The wall CI hit on day one#

The check in step 4 above failed for ten minutes with "last saw nothing". The site was serving the right release; the CI runner was being handed Cloudflare's "Just a moment…" challenge page, because Bot Fight Mode treats a datacenter address as a bot. The fix was one switch. The more useful change was to the script: it now reports the HTTP status and the first bytes of what it received, instead of collapsing a challenge page, a 404 and a dead tunnel into the same word. A failure message that cannot distinguish its causes is a bug in the failure message.

What I would still fix#

1. Build the failover Worker. It is the one gap in the table above.
2. Alert from outside the house. In-cluster monitoring goes dark with the cluster; a probe on a Worker cron does not.
3. Move content authoring into the CMS and publish through it, so that the pipeline above is triggered by a publish rather than by a commit.
4. Cache the HTML at the edge. Today every page view reaches the homelab; the cache purge on publish is already in place.