What coverage cannot tell you
Line and branch coverage answer one question: did the tests execute this code? They say nothing about whether any test would notice if the code were wrong. A suite can run every line of a derivation and assert only that it returned an array.
Mutation testing asks the second question directly. A tool (Stryker, here) makes small changes to the source one at a time: it flips a comparison, removes a + 1, swaps && for ||. Then it runs the tests against each mutant. A mutant that no test kills is a change in behaviour your suite would have shipped.
The study data derivations
Until this week nothing mutation-tested the SDTM, ADaM and table derivations behind the study data pipeline. They sat under the library-wide coverage floor of 70% of branches, which is a low bar for code that decides who counts as having a treatment-emergent adverse event.
The first run scored 80.65% over 1,891 mutants. The spread matters more than the total. Each file now has its own floor at its measured score: the ADaM core and the table filters at 91, the Kaplan-Meier code at 88, and the table assembly at 56. That last number is the honest one, and it is written into the configuration rather than averaged away.
Writing tests aimed at the surviving mutants added 27 edge cases on hand-made SDTM fixtures: baseline ties on the first-dose day, age-group boundaries, overlapping analysis windows, year-only partial dates, censoring for a subject with no contact at all. Branch coverage on the ADaM core went from 82% to 96%, and on the event derivations from 76% to 91%.
A ratchet, not a target
The issue asked for 95% branch coverage across the module. Measured coverage after the new tests was 86.8%, and most of the remaining branches are ?? [] fallbacks that the public API cannot reach. Tests written to hit them would exercise paths that real inputs never take, only to move a number.
I chose a ratchet instead. The floor is set at today's measurement (96% of statements, 86% of branches, 99% of functions and 96% of lines), and a test pins it so it can only go up. A target says where you wish you were. A ratchet says where you are and does not let you slide back.
The mutant that was a data problem
An independent QC suite programs the key derivations a second time from their specifications and compares them with the pipeline across 200 generated seeds. One mutant survives it anyway: move the end of the treatment-emergent window from fourteen days after last dose to thirteen, and nothing fails.
The assertions are not the problem; the QC logic for that boundary is in place. The synthetic generator never lands an adverse event on that boundary, never writes a partial end date, and never discontinues a subject for any reason other than an adverse event. No assertion can catch a case the data never contains.
That was the most useful thing the mutation run said. It points the fix at the test data rather than the tests: a library of deliberately messy scenarios, each with the findings it is expected to produce, is where those cases will come from.
What it costs
Mutation testing is slow, and the cost decides where it runs. The study data scope took 119 minutes on a loaded four-core container, so it runs weekly with a 150-minute bound, not on every pull request. The independent QC suite stays out of the mutation run: it costs about 30 seconds per re-import, which pushed the estimate past five hours.
The tooling can fail too, and this time it failed loudly. Upgrading to Vitest 5 broke the Stryker runner: it built test-name filters with spaces, Vitest 5 joins test names with " > ", and every mutant ran zero tests. The gate scored 0%. A zero is at least obviously wrong. A filter that matched half the tests would have produced a plausible number, and I would have believed it. Once the runner was updated, the pull request gate came back at 82.24%, the same as before the upgrade, which is the check that the upgrade changed nothing it should not have.
When to bother
Most code does not need this. Derivations do, along with anything where a wrong answer looks like a right one: a count that is off by the subjects on a boundary day still renders as a perfectly good table. For that kind of code, coverage tells you the tests were there. Mutants tell you they were paying attention.