Skip to content
Skip to main content
Novel Systems home
Decision log
D-086August 5, 2026

Publish the result, not the gate, and fingerprint the tree it measured

Decided

the backend suite is run, and its real output — 409 tests, 50 suites, per-file coverage — is committed to public/coverage.json and rendered on /engineering. A new check:coverage in the verify chain refuses to let the artifact drift from the code it describes.

Affects: scripts/build-coverage-report.mjs, public/coverage.json, lib/engineering-evidence.ts, lib/engineering-practice.ts, app/engineering/page.tsx, scripts/check-links.mjs, package.json

What went wrong

/engineering published the three coverage gates (lines 90, functions 90, branches 80), read off the flags on the backend's own test command, and the list of test files. It published no achieved figure, no test count, and no fetchable artifact. A reviewer read it and wrote: "no visible coverage report — no percent badge, no published coverage artifact, no public table saying which modules are covered and to what extent," and separately, "there's a difference between tests existing in the repo, and tests being visible as a public credibility artifact."

They were reading the page correctly. A gate is a promise about a number. The number was nowhere on this site, and a promise about an unpublished number is the kind of claim this page exists to stop making.

This is the fourth instance of one failure. D-069: the OpenAPI document existed at an address nobody guesses. D-070: the migration counts were a summary of a document that could not be opened. D-071: the spec could be fetched but not read. Now the test run — generated, correct, passing, and unfetchable. The fix has been the same every time, and it is worth naming so it stops being rediscovered: publish the bytes.

The alternative that was rejected

Two of them.

Running the suite during next build. It would produce a fresher number and it is the wrong place: the frontend build has no business executing the backend's tests, the tests need tsx and a module-mocking flag that has nothing to do with rendering pages, and a Vercel build that fails because a backend unit test failed is a confusing way to learn that. The house pattern — public/openapi.json and public/schema.sql — is generate-and-commit, checked by a guard. This follows it.

A third-party coverage badge (Codecov, Coveralls). Both want the repository, and this one is private; a badge for a repository the reader cannot open is a logo, not evidence. Publishing our own artifact is not more trustworthy in principle — but it is checkable in a way an image is not.

What was done instead

scripts/build-coverage-report.mjs generate runs npm --prefix backend run test:coverage, parses node's coverage table back into paths by reading its indentation, records the counts and the gates, and writes public/coverage.json. If the run fails, it prints the tail of the output and writes nothing: publishing coverage for a failing run is worse than publishing none, because the percentage looks identical to an honest one.

check mode, chained into verify and prebuild, fails if the gates in the artifact no longer match the flags on the live command, if any achieved figure sits below its own gate, if the module list has changed, if the run had a failing test, or if a SHA-256 over the 66 files of backend/src, backend/test and backend/package.json no longer matches the digest recorded with the run. That last clause is the load-bearing one: it means the published percentage cannot outlive the code it measured. Edit one backend file and the build fails until the suite is re-run. This was verified by appending a comment to backend/src/lib/errors.ts — check:coverage exited 1 — and reverting it.

/engineering now renders achieved against gate for all three dimensions, 409/409 passed across 50 suites in 9.1s, the runner and command by name, a 48-row per-file table, and the digest, with a link to /coverage.json so the table can be checked against its source.

The honest paragraph, and why it is on the page

The reviewer also asked for "a public CI/test log trail." That one cannot be answered: the repository is private, and a GitHub Actions link would 404 for precisely the reader who wanted to open it. The page says so, in those terms, rather than shipping a link that fails on click. What stands in for the log is the artifact plus the digest — which does not prove the run was honest, and the page says that too. Nothing published by the party being assessed proves its own honesty. It proves the numbers came from a named command against a fingerprinted tree, and that neither has moved since.

What would have to change for this to be wrong

If the repository became public, a link to the workflow run would be strictly better evidence than a committed file, and the honest paragraph above should be replaced with that link rather than kept alongside it. If the backend moved to a runner that emits LCOV, publishing the LCOV would be better than a bespoke JSON shape. And if check:coverage ever starts being satisfied by regenerating the artifact instead of fixing a failing test, the guard has become a ritual — the digest makes staleness fail loudly, but nothing stops a person from running coverage:generate to make a red build green. That is a discipline problem, not a tooling one, and it is recorded here because it is the way this decision most plausibly rots.

A smaller decision inside it

scripts/check-links.mjs kept a hand-maintained set of paths that are served out of public/ rather than rendered by a page — and it failed the build for /coverage.json because the new artifact was not in the list. The set had the failure mode backwards: it excused a link to a public file that had been deleted, and rejected a link to one that exists. Public-file links are now resolved by asking the filesystem, with the path normalised so a link cannot walk out of public/.