Sixteen ways to break a study
The study data pipeline on this site has an "Inject an issue" menu. Each of its sixteen scenarios changes the synthetic study in one way real study data gets messy: an adverse event that started on 30 February, a screen failure screened again under the same subject number, a vital sign entered twice, sex recorded as FEMALE instead of F, a lab unit that changes mid-study, a missing baseline, a death on treatment. Each scenario also declares which of the site's own checks should fire and what the derivations should do with the bad value.
The site's own checks flag or handle all sixteen as declared. That proves little, because they were written alongside the scenarios. So on 26 October I ran CDISC CORE, the open-source engine CDISC publishes for its conformance rules, over the clean run and every scenario, and committed what it found.
What CORE saw
On the clean run, CORE evaluated 430 SDTM rules. 282 passed, 136 did not apply to this study, 9 reported issues, and 3 hit engine errors. Every reported issue was already explained in the pipeline's list of known, justified findings.
Then the scenarios. For twelve of the sixteen, CORE's output was identical to the clean run. For the other four:
- Impossible date. CORE did not say the date was impossible. It said the adverse event had no epoch, which is empty because no epoch can be derived from a day that does not exist. Right row, indirect reason.
- Rescreened subject. Caught directly: the subject identifier is no longer unique in demographics, and the supplemental qualifiers are no longer unique per subject.
- Death during treatment and withdrawal before Week 12. A finding that already existed on the disposition record changed its value. Nothing new was reported.
The duplicated vital sign, FEMALE on a codelist that admits only F and M, the unit change and the missing baseline produced nothing at all.
That is not a flaw in CORE
CORE answers one question: does this dataset have the shape the standard requires? Are the required variables present, are the keys unique, do references resolve, do values sit where the implementation guide says they go? That question matters, and reviewers ask it first.
Whether a measurement was entered twice, whether a subject has a baseline, whether a date exists on the calendar: those are questions about this study's data, and a published standard cannot answer them. A dataset can pass every structural rule and still say something false about a patient.
So a clinical warehouse needs two layers, and neither replaces the other:
- Conformance to the published standard. Run the engine the standard's authors publish, pinned to a recorded version, rather than a home-grown copy of their rules. When it and your copy disagree, it is your copy that has to explain itself.
- Study-specific data quality. Rules written against the protocol and the data: duplicates, missing baselines, impossible dates, units that drift. This layer is yours to write, and nobody else can.
The twelve silent scenarios also say which of the site's own rules are worth keeping. A local rule that duplicates a CORE rule is a second implementation to maintain. A local rule that catches what CORE cannot is the whole point of having local rules. The next step of this work retires only the first kind.
The file that went stale in four minutes
CORE needs Python and several minutes per run, so it runs in a CI workflow, and its findings are committed to the repository as a JSON file the site can read with no server compute. Every view in that file carries a SHA-256 fingerprint of the pipeline output CORE ran on. A unit test recomputes the fingerprints and fails when any of them differ, so a stale file fails npm test in seconds, without Python.
That test paid for itself within the hour. A pull request that added the inclusion and exclusion domain for screen failures merged at 23:51 UTC. The CORE pull request merged at 23:55. Each had passed CI against a main that did not contain the other. Together, the pipeline now produced a domain the recorded findings had never seen, and the test failed for all seventeen views. The fix reran CORE and recorded again: two new findings, both already explained, and five rules that had skipped for want of that domain now passed.
Without the fingerprint, nothing would have failed. The file would have gone on describing a dataset that no longer existed, and the site would have shown CDISC's verdict on data CDISC never saw.
What I would take to any warehouse
- Plant defects, then see which layer catches each one. The gaps tell you what your own rules are for.
- Do not read silence from a conformance engine as clean data. It means the structure is sound, and nothing more.
- Record the engine's version with its results. A finding without the rule set that produced it cannot be reproduced.
- Treat every committed derived file as a cache. Store the fingerprint of its inputs beside it and test that the two still match. Two green pull requests are not a green merge when one generates a file from the other's inputs.
CDISC CORE is CDISC's open-source engine. Running it here does not imply CDISC endorsement.