Overview
This article is a dated build log for the 21 June 2026 Lab observability session. Since then, atlas-notify has gained named signal_class routing and atlas-api-index has become the fail-closed authority for the public Worker registry. The persist-only decision, browser-facing defects, and debugging lessons below remain accurate to the build described.
The Lab page at atlas-systems.uk/lab existed before this session, with a 90-day commit heatmap and a Failure log reading from a ring buffer in atlas-notify. Both worked. Neither showed the whole picture. The heatmap counted commits; the Failure log showed runtime events, GitHub pushes, Cloudflare alerts, the occasional pages_deploy from deploy-watch. Every CI/CD pipeline outcome across the rollout scope, every Worker deploy, every blocked validation, and every static-site publish stayed in Discord and nowhere else.
This piece closed that gap for the eight repositories then calling atlas-infra's reusable workflows. Their pipeline events began writing into the same ring buffer the Lab page already read, without changing the existing Discord notifications. Two new panels went up alongside it: a pipeline status grid showing the state of each rollout repository as a coloured tile, and an API surface map rendering the then-current endpoint declaration as worker-grouped cards.
Three real bugs surfaced building this. None of them were new. All three were sitting in already-shipped code, waiting for a consumer that actually looked.
Background
Atlas Pipeline built the event bus and the Lab page's first two panels. The CI/CD piece replaced five separate deploy.yml files with two reusable workflows shared across the rollout scope, gated and observable. Neither one wired the two together. The CI/CD rollout left a question open on purpose: should pipeline events flow through atlas-notify into the Failure log, or stay as direct curl posts from inside the reusable workflows. The deferral was deliberate; picking the wrong shape here meant either three Discord webhooks losing their separate channels, or a permanent gap in what the Lab page could show.
Two options were on the table going in. Route everything through atlas-notify, so it posts to Discord and writes to the ring buffer from one place. Or leave the direct curl calls untouched and accept that the Failure log only ever shows half the rollout. Neither was right.
The Persist-Only Decision
At the time of this build, atlas-notify held a single DISCORD_WEBHOOK_URL. The reusable workflows posted to three separate channels (api-deploys, deploy-log, ci-cd) by design, so a regression in one pipeline class stayed legible against the others. Routing everything through one Worker with one webhook would have required building multi-channel routing into atlas-notify just to get back to where the estate already was, plus a circular dependency: atlas-notify's own deploy would need to notify atlas-notify during the exact replacement window.
There was a sharper cost than the plumbing. Full routing would have made atlas-notify a single point of failure for every pipeline's Discord notification across the rollout scope, not just for the Lab page's history. A bad deploy to atlas-notify could have left every other repo's CI/CD blind in Discord, not only invisible on the Lab page.
Keeping the direct curl calls untouched and doing nothing else avoided all of that, at the cost of the Lab page never being a complete picture. That was the actual status quo going into this session, and it was the option I rejected, not because it was unsafe, but because it left the dashboard half-finished.
The resolution: keep the direct curl to Discord exactly as it already worked, and add a second, best-effort POST to atlas-notify carrying a persist_only flag. The Worker writes that event to the ring buffer and skips the Discord forward entirely. If atlas-notify is down or slow during a deploy, the Discord notification has already landed through the unaffected direct call; the only thing lost is one entry in a historical log, which is the same acceptable loss the ring buffer's own KV consistency note already accepts elsewhere.
Discord notifications and Lab page persistence are two separate calls, not one routed through the other. The direct curl is the guarantee; the persist_only POST to atlas-notify is the enhancement layer. One can fail without the other noticing.
atlas-notify later gained named signal_class routing for multiple destinations. That evolution changes the present-day routing capability, but it does not invalidate the June decision to keep the deploy notification and historical persistence paths independently failure-tolerant.
Wiring It Into Every Pipeline
Both of atlas-infra's reusable workflows, deploy-worker.yml and validate-static.yml, already ended every run with a notify job that built a Discord embed and posted it. The fix was a second step appended after that one, with continue-on-error: true, sending the same outcome through the new envelope shape with persist_only: true set.
The step checks for NOTIFY_TOKEN before doing anything and exits cleanly if it is not set. That mattered for the rollout itself: the shared workflow change could land independently from the per-repository secret setup, so repositories without the secret would skip persistence rather than fail their next pipeline run. Current production callers pin immutable atlas-infra revisions, so comparable workflow changes now propagate through explicit pin-update pull requests rather than floating into every consumer automatically.
Three of the four existing severity levels carry pipeline events: success for a clean deploy, warning for a cancelled run, failure for both a blocked validation and a failed deploy after validation passed, distinguished only by title text (Blocked: versus Deploy failed:). The fourth level, info, stays reserved for runtime events; a pipeline event is never neutral by definition, it either worked or it did not.
Setting NOTIFY_TOKEN across eight repos with gh secret set failed with a 403 on every call, the kind of error that reads like a permissions problem. The actual cause was a GITHUB_TOKEN environment variable set in the shell profile, left over from an unrelated work account, silently overriding gh's own stored credentials on every invocation. Prefixing each command with GITHUB_TOKEN= cleared it for that single call without touching the shell profile. gh auth status would have surfaced this immediately; the lesson was to check which identity a CLI tool is actually using before assuming the failure is about permissions rather than identity.
Phase I — The Wrong Eighth Repo
The pipeline status grid needed a fixed list of eight repo names to build its tiles from. Seven were unambiguous. The eighth was typed from memory against an earlier example payload rather than checked against the actual rollout scope, and landed as atlas-scheduler.
atlas-scheduler was not one of the eight repos wired to the reusable workflows; site-pulse was. The grid silently built a tile for a repo that would never emit an event, while a real repo's events had nowhere to land. An old alert that happened to mention atlas-scheduler from an earlier test bled onto a tile it had no real claim to.
The fix was a one-line swap in the grid's repo list. The orphaned alert did not need cleaning up separately; once the wrong tile stopped claiming it, it fell through correctly to the Failure log below, which is where an event with no matching repo belongs.
Repo list corrected to the real eight. No data was lost or needed recovering; the event was always landing in the right place underneath the grid, the grid was just reading it wrong.
Phase II — Two Versions of "Latest"
For repos with a successful deploy followed by a later routine push, the grid's status dot stayed correctly green, last deploy was good, but the one-line summary underneath it read "Push to main", the literal newest event regardless of severity. The two halves of the same tile told different stories.
The dot and the summary text were independently selecting "the relevant event" from the same event list using two different rules. The dot scanned newest-first and took the first event with a real severity, skipping informational pushes. The summary simply took the absolute newest event with no severity filter. Both selections were individually correct for what they were computing; they were just computing two different things and rendering them as if they agreed.
One shared selection feeding both halves of the tile: const summaryEv = status || latest. A green tile now reads "Deployed"; a red one reads "Deploy failed". The fix was a single line once the actual disagreement was identified, which was the harder part, since nothing threw an error and the page simply looked plausible while quietly lying.
Phase III — CORS Was Only Half Built
Backfilling synthetic deploy events for repos that had not pushed since the rollout meant POSTing to /notify from the browser console while looking at the Lab page directly. Every attempt failed at the preflight with CORS Missing Allow Origin.
atlas-notify's corsHeaders() function declared Access-Control-Allow-Methods: GET, OPTIONS only, with no Access-Control-Allow-Headers at all. The router's OPTIONS handling was scoped to a single path, /notify/recent, so a preflight for OPTIONS /notify matched nothing and fell through to a generic 405 carrying no CORS headers whatsoever. curl does not enforce CORS, so every server-side caller, including the GitHub Actions workflows themselves, had been working the entire time; only a browser-originated request was ever blocked.
The workaround for the immediate task was simple, run the synthetic backfill from a terminal instead. The underlying gap was worth fixing properly rather than working around again later. corsHeaders() was rewritten to allow GET, POST, OPTIONS, declare Authorization and Content-Type as allowed request headers, and set a 24-hour preflight cache. The single path-specific OPTIONS check was replaced with one global catch-all at the very top of the request handler, ahead of every other route, since a preflight can arrive for any endpoint, not just the one that happened to need it first. Every response inside the POST /notify handler, nine separate return points including the Discord-rejected 502 branch, was threaded with the same CORS headers, merged carefully with the existing Retry-After header rather than overwriting it.
Two new tests went in alongside the existing twelve: one confirming an allowed origin gets a 204 with POST present in the allowed methods, one confirming an untrusted origin gets no Access-Control-Allow-Origin echoed back at all. The same gap turned up a second time once the API surface map needed to fetch the plain GET / index cross-origin; the index endpoint had never carried CORS headers either, on either its JSON or HTML response branch. Same shape, two more lines, caught before it shipped rather than after.
CORS is now handled once, globally, for every method and every path this Worker serves, rather than added piecemeal per endpoint as each one happened to need it from a browser.
Closing the Surface
During this June build, the API surface map fetched the existing GET / endpoint, which returned a static JSON object hand-written into atlas-notify's source: a list of documented routes, methods, owning Workers, and one-line descriptions. Nothing was discovered or tested. The panel grouped that declaration by Worker and rendered each route as a card, with POST visually distinguished because it changes state.
Two endpoints carried (auth required) in their description text. Rather than leave that buried in a sentence, a short regex pulled it out into its own small tag next to the method badge, so a visitor skimming the page could see what was public and what was gated without reading every line of prose.
The panel fetched once on page load rather than polling on an interval the way the heatmap and Failure log did. That was deliberate: the data was a source declaration, not an event stream. It changed only when a Worker was redeployed, so polling it every thirty seconds would have spent requests against something that was almost always identical.
atlas-vault did not appear anywhere in that index. That was a boundary, not a gap; it was a personal backup vault, and there was no reason its surface needed to be public just because every other Worker's did. The map documented what was meant to be seen.
The current estate no longer treats that hand-authored atlas-notify object as the registry authority. atlas-api-index now owns fail-closed public Worker discovery, probing only explicitly approved services and publishing the public registry at api.atlas-systems.uk/. The old map remains useful as evidence of the first consumer-facing API inventory and of why a generated, policy-backed registry became necessary.
Outcomes
atlas-notify across the persist_only handler and the CORS fixPipeline events from every repository in the June rollout scope began showing up in the same place runtime events already did, without changing the shape or timing of the existing Discord notifications. Three real bugs were found and fixed in the process: a wrong repo name, two disagreeing event selections on one tile, and a CORS gap that had existed since the Worker was first written. None was introduced by this work, and all were invisible until a browser-facing consumer exercised code paths that had not previously been observed that way.
The deeper pattern worth keeping is that the engineering effort here was not new infrastructure. atlas-notify already existed. The ring buffer already existed. The first API declaration already existed as a static object nobody was rendering. The work was about building thin, honest views onto systems that already worked, and the bugs that surfaced were old defects seen from a new angle. A system can be correct and still be unobserved; the gap between those two states is often smaller, and more interesting, than building something from nothing.