“What is our coverage?” sounds like a question about quality. In practice it is a question about what happened to execute. When a model writes or reviews tests, a dashboard percentage is eager to become the answer. This post is about why that is a weak validator.
Neighbours, not substitutes
The time spent understanding a change before you even pick scope is in the hidden tax on every pull request. A pipeline that cries wolf is in noisy CI. Separately: a green e2e whose assertion was already true before the click. Here is another certificate people and agents like to treat as a verdict: the coverage percentage.
None of those signals is replaceable by a single “we have 80%” gate.
A short lab
A private demo stack, classic RealWorld split into an SPA and an API. Tests already existed. Nobody collected, published, or gated coverage.
When I measured this, e2e main had one critical path: unique registration, publish an article, see the preview on Global Feed. I did not add journeys to raise the number. That is the point of the setup, not a hole in the suite.
That leaves four reports. I am not folding them into one percentage for the whole stack.
On an application PR there is only unit coverage. Jest on the frontend, Coverlet on the API Mediator slice. Those two checks do not start Playwright. The Hurl spec stays a separate job.
E2e has its own harness and a COVERAGE flag. After an application PR merges, the same Node spec runs with a mutated API process (dotnet-coverage) and a V8 harvest. Twice a day, morning and evening, that same Playwright run hits an unmutated dotnet run. Those are not the same certificate.
Four numbers you must not add up
Each of these files has a different denominator and a different process. Do not add them up. Slice Coverlet looks at 546 lines of the Mediator slice. E2e API looks at 1110 lines of the live Conduit process. That is not the same fraction with a different numerator.
| Report | What was actually measured | Lines |
|---|---|---|
| Jest | Two dummy tests (true / false). Denominator: app src/ without the generated client |
0% (0/727) |
| Slice Coverlet | Existing Mediator tests on InMemory, no Kestrel, no HTTP | 41% (224/546) |
| E2e API | The same e2e-repo journey in Chromium, dotnet-coverage on the live process, then a filter to Conduit |
46% (512/1110) |
| E2e V8 | JS from that session, mapped to src/ from loaded chunks |
87% (1884/2155) across 43 files |
Branches: 32% on the slice, 21% on e2e API, 62% inside those 43 V8 files.
CI waits on GET /api/tags before Playwright starts. That is not the featured journey. The tags controller is green in the e2e report because the host had to answer, not because a user clicked tags.
A harness, not a second copy of the assertions
I am not duplicating the e2e suite in C#. Chromium is driven by the same Node spec from the e2e repo. The COVERAGE flag only changes the harness: plain dotnet run, or that process wrapped by a collector, plus V8 in the same browser session. I wrote earlier about WebApplicationFactory and DI and Blazor with Playwright. The tests stay in e2e. The mutation is on the process, not a second copy of the assertions.
Before those four files were even quotable, three artifacts lied. Coverlet on CI wrote an empty Cobertura document, and the job was green. The e2e API headline pulled in FluentValidation and dropped to 15%, even though the journey only went through Conduit. V8 harvested only after the last goto kept boot and home: register and editor vanished with the lazy chunks. The harness I kept fails an empty file, filters Cobertura down to the Conduit package, and harvests V8 before every navigation. The percentages in the table are from that run, not from the first attempt.
Coverlet on the Mediator slice does not see controllers. dotnet-coverage on the e2e run sees HTTP, because the request actually entered the pipeline. V8 does not see C#. C# does not see TSX. If you ask a model “what is our coverage?” and you only keep one badge, you have given it permission to glue those worlds together.
What this journey never loaded
Uncovered does not mean “bad.” It means “this path did not go there.”
On the API, at 46% of Conduit lines, hits stay at zero for login, profile edit, follow, favorite, comment create/delete, article edit/delete, and profiles, among others. The comment list is 90% anyway, because the article page fetches comments. Nobody posted a comment. A read can be “covered” while the write behaviour is missing.
On the UI, Jest says 0%, because the dummy tests do not import the app. V8 says 87% - but only across the 43 files webpack put into the chunks that loaded. Nineteen files from the same src/ never enter the report: login UI, 404 UI, profile, logout, settings save, comment create/delete, follow, favorite, article edit and delete.
login.route.ts is 100% in the V8 report. We did not log in through the form. The router imported the module at boot. settings.ui.tsx is also 100%, because registration lands on settings. Nobody saved anything and nobody logged out. The percentage does not distinguish “the file was imported or rendered” from “the user walked the path.”
What a model does with an 80% gate
It receives 87% from V8 and a green light, even though login UI is absent from the report. Or it receives 0% from Jest and starts “fixing coverage” where it is cheapest: more asserts on code that already ran, the generated client, vendors, another expect on the same preview tile.
That is why this lab keeps artifacts and has no threshold. A threshold turns an execution map into a pass certificate. An agent you ask to review tests will optimise the certificate.
You do not need to buy a coverage product to see the mechanism. Four files side by side, and a ban on adding them up, are enough.
Close
Keep the reports separate. The next decision is human: whether this path even asks the question you care about - and what it deliberately never executed.
Coverage is a map of what ran. It is not a verdict that the tests are good, and it is not a verdict that a model wrote or reviewed them well.