The problem
Clinical study data often needs to move from an EDC export into a standard analysis format. This project tackles that mapping: read ODM metadata, validate the incoming records, and organize them into SDTM domains. The tricky parts are the study-specific fields and the rules hidden in the details.
The approach: A SAX parser reads XML incrementally. ODM metadata defines Zod validation schemas, and a separate mapping layer builds SDTM records.
Reported project measurements: Sub-50MB peak memory during 2GB+ XML ingestion; 100% CDISC Controlled Terminology validation parity; 12x throughput acceleration over DOM-based parsers.
Checking values against controlled terminology is one kind of data rule. The platform demo's data rules and catalog page shows how rules like it, data contracts and metric definitions are recorded and owned across a whole warehouse, on synthetic studies.
How it works
A Node.js streaming SAX parser, wrapped in TypeScript async generators, reads multi-gigabyte XML files in chunks without loading the whole DOM into memory.
At runtime the tool reads the ODM MetaDataVersion and compiles Zod schemas from it, so a protocol amendment updates the schemas without a code change.
A declarative compiler turns raw item data into SDTM domains (DM, AE, VS, LB), derives study days (--DY) and checks values against controlled terminology.
How the pieces connect
flowchart TD
A[Raw CDISC ODM XML: 2GB+] --> B[Streaming SAX Lexer & Parser]
B --> C[Async Generator Object Stream]
C --> D[Dynamic Zod Schema Synthesizer]
D --> E[Canonical Study Model Store]
E --> F[Declarative SDTM Domain Compiler]
F --> G1[DM: Demographics Domain]
F --> G2[AE: Adverse Events Domain]
F --> G3[VS: Vital Signs Domain]
G1 & G2 & G3 --> H[CDISC Controlled Terminology Validator]
H --> I[FDA-Ready SAS Transport / CSV Artifacts]
Implementation notes
Streaming SAX XML Parser & Generator (src/parser/odm_stream.ts)
// Streaming SAX Parser for Multi-Gigabyte CDISC ODM XML
import sax from "sax";
import { Readable } from "stream";
interface SubjectDataChunk {
studyOID: string;
subjectKey: string;
items: Array<{ itemOID: string; value: string }>;
}
async function* parseODMStream(xmlStream: Readable): AsyncGenerator {
const parser = sax.createStream(true, { trim: true });
let currentSubject: Partial | null = null;
for await (const chunk of xmlStream) {
parser.write(chunk);
// Emits structured subject events as XML tags resolve
if (currentSubject && currentSubject.items && currentSubject.items.length > 0) {
yield currentSubject as SubjectDataChunk;
currentSubject = null;
}
}
}
Declarative SDTM Domain Compiler (src/compiler/sdtm_builder.ts)
// Declarative SDTM Domain Transformation Engine
function deriveStudyDay(eventDate: Date, referenceDate: Date): number {
const diffTime = eventDate.getTime() - referenceDate.getTime();
const diffDays = Math.floor(diffTime / (1000 * 60 * 60 * 24));
// SDTM convention: Day 1 is baseline, Day -1 is prior day (no Day 0)
return diffDays >= 0 ? diffDays + 1 : diffDays;
}
Tradeoffs and lessons
- Parsing 2GB XML files into a DOM crashed Node.js with out-of-memory errors. SAX streaming keeps memory < 50MB whatever the file size.
- SDTM has no Day 0, so the date math steps from Day -1 straight to Day 1 at baseline.