Why DOM parsers run out of memory
Electronic Data Capture (EDC) systems export large XML files in the CDISC Operational Data Model (ODM) format. Once a study reaches hundreds of sites and tens of thousands of subject records, a DOM parser that builds the whole document as an in-memory tree gets into trouble: multi-gigabyte heaps, long garbage collection pauses, latency spikes, and serverless workers that crash under peak load.
In a clinical study a failed ingestion costs more than an operations ticket. A failed extract delays monitoring audits, holds up interim safety reviews, and puts database lock deadlines, which are set by submission windows, at risk.
Streaming the XML instead
An event-driven SAX parser handles the study one record at a time. The tokenizer walks through the document and emits a typed event each time an element opens or closes, so the whole archive never sits in memory.
Bounded memory
Each study event, form definition, item group and item compiles straight into a typed domain object. The heap stays under 50 MB whether the incoming file is 5 megabytes or 5 gigabytes:
interface ItemDataRecord {
itemOID: string;
value: string;
transactionType?: "Insert" | "Update" | "Upsert";
auditRecord?: {
userRef: string;
dateTimeStamp: string;
reasonForChange?: string;
};
}
interface ClinicalSubjectStream {
studyOID: string;
metaDataVersionOID: string;
subjectKey: string;
items: AsyncIterable<ItemDataRecord>;
}
Because the pipeline is a stream, a worker can push back on the upstream broker whenever the database starts throttling writes, so buffers never pile up.
Mapping to SDTM
The FDA and PMDA require CDISC SDTM for human drug submissions. Turning operational EDC extracts into tabulation domains (such as Demographics DM, Adverse Events AE and Vital Signs VS) takes several normalization passes:
- Check coded values against the quarterly NCI EVS codelists, pinned to one snapshot version.
- Turn partial dates into valid ISO 8601 character values for SDTM without silently truncating them.
- Hash the inputs to every derived observation with SHA-256, Part 11 style, so a reviewer can check where a record came from and spot tampering.
Provenance for every record
Database transaction logs are volatile, so the pipeline doesn't rely on them. Every transformed observation carries a hash that links the tabulation record back to its source ODM XML token offset and to the version of the rule that produced it:
interface ProvenanceCertificate {
sourceOID: string;
mappingRuleId: string;
ruleVersion: string;
sourceInputHash: string; // SHA-256 digest of raw ODM token
compiledAtUtc: string;
}
Load test results
In an automated load test against a 1.2 GB CDISC ODM study export with 45,000 subject visits, the streaming pipeline finished the whole SDTM transformation in 14.2 seconds on a single vCPU container, and memory peaked at 38.4 MB. The old DOM parser crashed on the same workload with an unrecoverable heap exhaustion error after allocating 2.8 GB.
The Cadence Clinical case study and the Clinical Data Mapper architecture show the boundaries that keep the audit surface small. To watch form rules compile as you edit them, open CRF Studio.