Shaped like the standard
CRF Studio builds case report forms and exports them as CDISC ODM XML, the format EDC systems use to exchange study metadata. It also exports a USDM study design. The study data pipeline writes an ARS reporting event that describes its analysis results. All of those files looked right. The element names matched the specifications, the structure was plausible, and our own importer read every file back without complaint.
Then I checked every preset's ODM export against the official XSD that CDISC publishes. Every one failed, for four reasons:
- Item definitions carried a
CodeListOIDattribute where the schema expects aCodeListRefchild element. - An annotated-CRF element sat inside the item definition, where the schema does not allow it.
- NCI codes were written as an attribute that neither ODM nor Define-XML defines.
- The design audit trail was written as ODM elements that do not exist.
Nothing had caught these because the only reader of the files was our own importer, written alongside the exporter. A round trip proves the two halves agree with each other. Whether either agrees with the standard is a separate question.
Use the schema the authors publish
Each fix was small (#3039). The audit trail moved into a CRF Studio namespace in ODM's extension slot, which is how Define-XML extends ODM, with its own small extension schema. The importer still reads files saved before the change. The test is the part that lasts: every preset is now checked against the vendored official XSD on every run, so the next drift fails CI instead of somebody's import.
The ARS reporting event got the same treatment (#3045). Checked against the official ARS JSON schema, it failed in five ways. Lineage fields sat on classes that do not define them. Some analyses had no purpose. Nested where clauses had no level or order. Two file types were outside ARS's enumeration. Results carried ids ARS has no place for. The lineage was worth keeping, so it moved to its own file beside the ARS file, and a test proves the two files together rebuild the whole event. The schema test runs over the default study, two other seeds and all sixteen messy-data scenarios.
Codes that looked right
Controlled terminology is where plausible and wrong are hardest to tell apart, because a C-code is a letter and some digits. I swept the 213 NCI codes cited in the site's code against the published NCI EVS release of CDISC terminology (#3050). Some were wrong in ways nobody would catch by eye:
- The USDM screening and follow-up epochs used general NCI Thesaurus concepts instead of the DDF epoch type terms.
- The code given for the NYHA class codelist belonged to a different codelist, and the PHQ-9 codelist code belonged to another questionnaire.
- One concept that exists only in the NCI Thesaurus was filed under the CDISC code system.
The gate is a script that reads the hash-pinned EVS exports and writes a subset of only the codes the repository cites: 34 codelists and 186 concepts. A test fails if a cited code is not published, if the subset holds a code nothing cites, or if a CRF Studio term is not a term of its own codelist. EVS answers a request for a missing archive with an HTML page instead of an error, so the script also refuses HTML.
Then the rules engine
A schema checks shape. CDISC CORE, the open-source conformance engine CDISC publishes, checks rules on top of it. An earlier post covered running CORE over the study data pipeline. On 27 October we ran its USDM rules over every CRF Studio preset's export as well (#3055). It reported 103 findings, which came down to 19 explanations.
Three of the nineteen were CORE's own mistakes: a rule that flags next and previous chains that are correct, one that reports a study's only title twice, and one that reports governance dates the export does not contain. Those are recorded as engine limitations, each with its reason. Bending the export to quiet them would have made it worse. The other sixteen were real export defects, among them missing eligibility criteria and five of the six presets with no arms, epochs or cells.
A follow-up fixed the ones the export code caused (#3061). The sponsor is now named as a study role. Every visit falls in an epoch, an unscheduled one in the epoch of the visit before it. The timeline orders visits the same way the visits order themselves. A one-arm design is labelled single group instead of parallel. That removed seven of the sixteen allowlist entries and narrowed two more. What remains is content CRF Studio does not hold yet, such as eligibility and endpoints, and the allowlist names each gap.
Check the checker too
The three misfires are the part I would point to when someone asks how to evaluate a tool, from a vendor or from your own team. An outside reference is useful only if you read what it says and are willing to disagree with it in writing. A suppressed finding needs a reason a reviewer can read. A finding you cannot explain needs a fix.
What I would take to any team that exports a standard:
- Get the official schema and run it in CI. A round trip through your own importer proves consistency with yourself, and that is all.
- Pin every reference: the schema, the terminology release, the engine version. A result can be reproduced only with its inputs.
- Check terminology codes against the published release, not a list someone typed once.
- Treat the reference engine as a second opinion to evaluate. Record its false positives with reasons, and fix everything else.
CDISC CORE and these schemas are CDISC's. Using them here does not imply CDISC endorsement.