A badge I should not have shipped
This week I had my own site reviewed the way a clinical data manager would read it, and I did not like what came back. CRF Studio carried a badge announcing one hundred percent conformance with CDISC and SDTM. Its audit log was described as an unalterable Part 11 trail. A status line declared the whole thing ready for regulated work.
None of that was implemented. What existed was a useful subset of design checks from the CDASH and SDTM implementation guides, an audit trail that lives in browser state, and a local slice of controlled terminology. That is a reasonable thing for a portfolio to demonstrate. It is not what the copy said, and the people the clinical work is written for are exactly the readers who notice the difference and stop trusting the rest of the page.
I cannot quote the old wording here. This file is one of more than 130 that a test now scans for it, and the build would fail.
Describe what is implemented
The replacement rule is easy to state: copy describes the checks that exist, never the standard they are aimed at. A clean run now reads "No findings from the implemented checks." The audit trail is a "Part 11-style audit trail (simulation)". The terminology is a local subset aligned to one NCI EVS package.
A rule like that decays the moment nobody is watching, so it is a function rather than a guideline. findClinicalClaims() matches the shapes that kept appearing: a standard or regulation followed by "compliant", a percentage score in front of a standard, a promise to comply with a regulation, an audit trail that cannot be changed. A test runs it across the clinical routes, components and libraries, the social card presets and this blog, and names the phrase and the file when it fails. It also has to let honest sentences through, so the test keeps a list of those as well. "Three non-compliant records." must pass.
Then I checked the codes
"Aligned to the package" was itself a claim, so the next job was to check it. Every codelist and term in CRF Studio went up against the NCI EVS SDTM terminology file for the package date the site declares.
It was worse than the issue I had filed. Most term codes were wrong or missing from the package, and several codelists sat under codes that belong to other codelists:
- The yes/no codelist used C66741, which is the vital signs test code list. Sex used C66742, which is the yes/no list.
- Race and ethnicity pointed at codelists that have been retired; their last appearance was the 2025-09-26 package. Collection now uses C128689 and C128690.
- Severity was filed with CTCAE grade codes. Mild, moderate and severe have their own: C41338, C41339 and C41340.
These were not cosmetic. They were published in ODM def:NCICode attributes, SAS format comments, the annotated CRF and the Word export. Every one of them looked right, which is the worst property a wrong identifier can have. A misspelled term fails a lookup. A real code from the neighbouring codelist passes every check that does not go back to the source file.
The error had spread, too. The study data pipeline wrote a dose unit of TABLETS, an extension that existed only to match the wrong CRF code. It now writes the package's own TABLET, C48542, and the golden files that changed changed for that reason alone.
An outside opinion, in CI
My own conformance stage checks my own reading of the rules. That is circular in the way that matters, so the pipeline's outputs now also go through CDISC's open-source rules engine, CORE, pinned to v0.17.1, in a separate workflow that runs whenever the pipeline changes and once a week.
The first run reported findings on 69 rule and dataset pairs. Sorted honestly:
- 55 are real defects in my pipeline or its study definition, each tracked in an issue.
- 12 are engine limitations, such as duplicate-check rules that misfire on the fixture.
- 1 is by design: adverse events carry no epoch.
- 1 is a gap in the engine's terminology cache.
The design decision worth copying is the allowlist that explains them. CI fails on any finding the list does not explain, which is the usual rule. It also fails on any entry that CORE no longer reports. A fix proves itself by making its allowlist entry stale, and the list cannot quietly become a museum of problems solved months ago.
The job summary also says what was not checked. CORE's newest cached terminology is from March while the pipeline declares September, so the export substitutes the older release and says so. CORE has no ADaMIG rules yet, so the analysis layer is reported as "not run". The workflow asks the engine for its rule list rather than hardcoding that, so it will notice when they arrive.
Program it twice
Clinical programming has an old answer to "how do you know the derivation is right": a second programmer writes it again from the specification, without looking at the first, and the outputs are compared. The pipeline now has that, as a test suite. Study days, baseline selection, change from baseline, last observation carried forward, adverse event date imputation, treatment-emergent flags, time to event and the three summary tables are each derived again from the specification and compared row for row with the pipeline, on the study's own seed and on 200 generated ones.
Its pull request listed what it could not test. The synthetic generator never lands an adverse event on the edge of the treatment-emergent window, so a mutant that moves that edge from fourteen days after last dose to thirteen survives. That gap is written down next to the suite instead of being left for a reader to find.
Smaller claims, more checks
The page that runs the pipeline now says what it is: one fictional study, synthetic subjects, and a conformance stage that models a subset of published rules and is not a validator of record. That is a much smaller sentence than the badge was. Behind it sit a wording guard, a terminology check against the source file, an outside rules engine whose findings are tracked one by one, and an independent second implementation.
I would rather publish the small sentence with all of that behind it than the large one with nothing.
You can run the pipeline and download its package, or read how it is built in the Study Data Pipeline case study. CRF Studio is where the corrected codes now live.