TL;DR
- Fragment processing breaks complex FHIR resources (Encounter, Patient, Observation) into smaller, domain-aligned units. One resource can generate several fragments: a blood pressure Observation, for example, becomes three measurement rows.
- The pipeline runs in 4 stages: Fragment Generation → Parallel Processing → Staging (TSV) → Reduction.
- Fragments from one export are generated and reduced in parallel across compute nodes, so throughput grows with the cluster instead of with a single process.
- Row-by-row ETL handles one record at a time. Fragment-based parallelization splits the same work across nodes.
Fragment processing converts FHIR resources into the OMOP Common Data Model (CDM) by splitting each resource into pieces that can be processed in parallel. At Mindbowser, we build FHIR-to-OMOP pipelines on this fragment-processing pattern. ConnectHealth, our EHR integration product, can supply the input: FHIR bulk exports from Epic, Oracle Health and other EHRs, running in your own cloud.
These fragments are enriched with standardized terminology, deduplicated, and efficiently bulk‑loaded into OMOP tables. This “fragment‑processing” approach solves key challenges of scale, data integrity, and performance, enabling healthcare organizations to transform raw EHR data into analytics-ready repositories with speed, accuracy, and compliance.

The Performance Challenge
Healthcare systems often struggle with fragmented data across EHR systems, making efficient, high-quality transformation of millions of complex medical records a major challenge. Traditional row-by-row processing becomes a bottleneck when dealing with Epic’s bulk exports containing hundreds of thousands of patient encounters.
Fragment-based processing breaks complex transformations into parallelizable components.
Understanding Fragment Architecture
Fragment processing operates on a simple but powerful principle: instead of trying to transform complete FHIR resources into final OMOP records in a single step, break the process into manageable pieces (fragments) that can be processed independently and then intelligently assembled.
The Fragment Lifecycle
- Fragment Generation: FHIR resources are decomposed into target-table-specific fragments
- Parallel Processing: Multiple fragments are processed simultaneously across compute nodes
- Staging: Fragments are written to disk in an efficient tab-separated format
- Reduction: Fragments are consolidated, conflicts resolved, and final records generated
This approach enables massive parallelization while maintaining data integrity and clinical context.
Stage 3: Fragment Generation Deep Dive
The fragment generation stage is where most of the transformation work happens. Stages 3 and 4 run as part of the broader medallion pipeline covered in FHIR-to-OMOP on Databricks. This page focuses on the fragment-level mechanics inside those stages.
Example: Complex Encounter Processing
A bulk export delivers each resource type in its own file. An Encounter, a Condition recorded during it and a blood pressure Observation arrive as three separate resources, linked by reference.
Input (NDJSON, one resource per line):
{"resourceType": "Encounter", "id": "encounter-12345", "status": "finished", "class": {"system": "http://terminology.hl7.org/CodeSystem/v3-ActCode", "code": "AMB"}, "type": [{"coding": [{"system": "http://snomed.info/sct", "code": "185349003", "display": "Encounter for check up"}]}], "subject": {"reference": "Patient/patient-67890"}, "period": {"start": "2023-12-15T10:00:00Z", "end": "2023-12-15T11:00:00Z"}}
{"resourceType": "Condition", "id": "condition-555", "code": {"coding": [{"system": "http://snomed.info/sct", "code": "59621000", "display": "Essential hypertension"}]}, "subject": {"reference": "Patient/patient-67890"}, "encounter": {"reference": "Encounter/encounter-12345"}, "onsetDateTime": "2023-12-15"}
{"resourceType": "Observation", "id": "obs-777", "status": "final", "code": {"coding": [{"system": "http://loinc.org", "code": "85354-9", "display": "Blood pressure panel"}]}, "subject": {"reference": "Patient/patient-67890"}, "encounter": {"reference": "Encounter/encounter-12345"}, "effectiveDateTime": "2023-12-15T10:15:00Z", "component": [{"code": {"coding": [{"system": "http://loinc.org", "code": "8480-6", "display": "Systolic blood pressure"}]}, "valueQuantity": {"value": 142, "unit": "mm[Hg]"}}, {"code": {"coding": [{"system": "http://loinc.org", "code": "8462-4", "display": "Diastolic blood pressure"}]}, "valueQuantity": {"value": 91, "unit": "mm[Hg]"}}]}Generated fragments (tab-separated in production; shown here with labels):
visit_occurrence encounter-12345 patient-67890 standard concept: Outpatient visit 2023-12-15 10:00 2023-12-15 11:00
condition_occurrence condition-555 patient-67890 standard concept: Essential hypertension source: SNOMED 59621000 visit: encounter-12345 2023-12-15
measurement obs-777 patient-67890 standard concept: Blood pressure panel source: LOINC 85354-9 visit: encounter-12345
measurement obs-777-sys patient-67890 standard concept: Systolic blood pressure source: LOINC 8480-6 142 mm[Hg] visit: encounter-12345
measurement obs-777-dia patient-67890 standard concept: Diastolic blood pressure source: LOINC 8462-4 91 mm[Hg] visit: encounter-12345In the reduction stage, each source ID becomes an OMOP integer key and the condition and measurement rows link to the visit through visit_occurrence_id.
Generated Fragments:
Fragment 1 (Visit Domain):
visit_occurrence encounter-12345 patient-67890 4024660 2023-12-15 2023-12-15T10:00:00 2023-12-15 2023-12-15T11:00:00 44818518 …
Fragment 2 (Condition Domain):
condition_occurrence encounter-12345 patient-67890 320128 2023-12-15 2023-12-15T10:00:00 2023-12-15 2023-12-15T11:00:00 32817 …
Fragment 3 (Procedure Domain):
procedure_occurrence encounter-12345 patient-67890 4301351 2023-12-15 2023-12-15T10:00:00 2023-12-15 2023-12-15T11:00:00 32817 …
Fragment Structure Design
Each fragment follows a consistent structure:
- Table Name Prefix: Identifies the target OMOP table
- Primary Key: Enables duplicate detection and merging
- Tab-Separated Values: Optimized for bulk loading performance
- Complete Row Data: All required OMOP CDM columns
This structure enables efficient downstream processing while maintaining full data integrity.
Parallel Processing Benefits
Parallel processing helps overcome the challenges of fragmented data by enabling the simultaneous transformation of independently structured FHIR resources.
Fragment generation enables massive parallelization:
Compute Distribution
- Multiple FHIR files processed simultaneously
- Independent fragment streams avoid processing bottlenecks
- Resource-specific processing optimizes for data characteristics
- Auto-scaling clusters adjust to processing demands
Memory Optimization
- Streaming processing avoids loading entire datasets
- Fragment caching optimizes repeated access patterns
- Garbage collection minimizes memory pressure
- Disk spilling handles datasets larger than memory
Talk to Our Data-engineering Team About Your FHIR-to-OMOP Pipeline
Stage 4: Fragment Reduction Mastery
The fragment reducer is where individual fragments are consolidated into final OMOP tables. This stage handles the complex logic of merging related data while resolving conflicts and maintaining referential integrity.
The reduction stage consolidates fragmented data generated across different domains, resolving conflicts and creating unified OMOP-compliant records.
🔹 The Reduction Algorithm
- Fragment Grouping: Group fragments by target OMOP table
- Primary Key Sorting: Sort fragments by primary key for efficient processing
- Conflict Resolution: Merge fragments with identical primary keys
- Quality Validation: Ensure final records meet OMOP CDM requirements
- Bulk Loading: Generate final tab-separated files for database import
🔹 Conflict Resolution Logic
When multiple fragments contribute to the same OMOP record, intelligent conflict resolution ensures data quality:
🔸 Non-Null Preference
Fragment A: person_id=123, gender_concept_id=8507, birth_year=NULL
Fragment B: person_id=123, gender_concept_id=NULL, birth_year=1975
Result: person_id=123, gender_concept_id=8507, birth_year=1975
🔸 Conflict Detection
Fragment A: person_id=123, gender_concept_id=8507
Fragment B: person_id=123, gender_concept_id=8532
Result: ERROR – Gender conflict for person 123
🔸 Source Preservation
All fragments maintain source_value fields for traceability:
condition_source_value=”38341003″ // Original SNOMED code
Performance Characteristics
What drives throughput:
🔹 Processing Speed
- Linear scaling with additional compute nodes
- Burst processing handles monthly bulk exports efficiently
In one engagement, a genomics research platform running a multi-year bulk FHIR pipeline used this approach to bring its OMOP transformation timeline from 11 months of custom ETL to a 3-week implementation with no loss of clinical fidelity.
🔹 Resource Efficiency
- Memory-optimized: Processes datasets larger than available RAM
- Disk-efficient: TSV format minimizes storage requirements
- Network-optimized: Minimizes data movement between nodes
- Cost-effective: Auto-scaling reduces idle compute costs
Quality Assurance Framework
Fragment processing includes comprehensive quality controls:
🔹 Fragment Validation
- Schema compliance: Each fragment matches target table structure
- Data type validation: Ensures numeric, date, and text field accuracy
- Required field checking: Validates presence of mandatory OMOP fields
- Concept validation: Confirms all concept_ids exist in vocabularies
🔹 Reduction Quality Checks
- Completeness monitoring: Tracks record counts through each stage
- Conflict reporting: Identifies and reports data inconsistencies
- Referential integrity: Validates foreign key relationships
- Statistical validation: Compares input/output record distributions
Monitoring and Observability
Production fragment processing requires comprehensive monitoring:
🔹 Real-Time Metrics
- Processing throughput: Records per second by stage
- Error rates: Fragment validation failures
- Resource utilization: CPU, memory, and disk usage
- Queue depths: Processing backlogs by resource type
🔹 Quality Dashboards
- Data completeness: Percentage of required fields populated
- Concept coverage: Vocabulary mapping success rates
- Processing latency: End-to-end pipeline timing
- Cost tracking: Compute resource consumption
Troubleshooting Common Issues
Fragment processing systems require proactive issue management:
🔹 Performance Bottlenecks
- Memory pressure: Optimize fragment size and caching
- Disk I/O limits: Distribute processing across storage systems
- Network bandwidth: Minimize cross-node data movement
- CPU saturation: Scale compute clusters appropriately
🔹 Data Quality Issues
- Unmapped concepts: Expand vocabulary coverage
- Duplicate records: Enhance primary key generation logic
- Missing references: Improve foreign key resolution
- Schema violations: Update fragment validation rules
Advanced Optimization Techniques
Production implementations benefit from advanced optimizations:
🔹 Intelligent Caching
- Vocabulary caching: Pre-load concept mappings for speed
- Fragment templating: Reuse common record structures
- Checkpoint optimization: Enable fast recovery from failures
- Result caching: Avoid reprocessing unchanged data
🔹 Adaptive Processing
- Dynamic clustering: Adjust resources based on workload
- Intelligent partitioning: Optimize data distribution
- Priority queuing: Process critical resources first
- Load balancing: Distribute work across available nodes
Future Enhancements
Fragment processing continues to evolve:
🔹 Real-Time Processing
- Stream processing: Handle real-time FHIR updates
- Incremental reduction: Update records without full reprocessing
- Change data capture: Track modifications to source data
- Event-driven architecture: Trigger processing based on data changes
🔹 Machine Learning Integration
- Intelligent conflict resolution: Learn optimal merge strategies
- Quality prediction: Identify potential data issues proactively
- Performance optimization: Automatically tune processing parameters
- Anomaly detection: Flag unusual data patterns for review
Fragment processing represents a fundamental advancement in healthcare data transformation, enabling the scale and performance required for modern Epic-to-OMOP pipelines while maintaining the data quality essential for research excellence.
By turning fragmented data from bulk FHIR exports into high-integrity OMOP records, fragment processing lays the foundation for scalable and trustworthy real-world evidence generation.
Fragment processing techniques build upon distributed computing principles adapted for healthcare data, with recognition to Carl Anderson and the FHIR Analytics community for pioneering scalable approaches to clinical data transformation.
How Fragment Processing Connects to FHIR-to-OMOP Analytics
Fragment processing is a key step inside the broader FHIR-to-OMOP analytics workflow. It helps break large, complex FHIR resources into smaller, domain-aligned components that can be validated, transformed, and loaded into OMOP tables with greater consistency.
This matters because healthcare analytics depends on trustworthy longitudinal data. If fragmented FHIR resources are not handled correctly, teams can create incomplete patient timelines, duplicate records, terminology mismatches, or gaps between source data and OMOP outputs. A strong fragment-processing approach preserves clinical meaning while improving processing speed, traceability, and downstream data quality.
In a mature FHIR-to-OMOP pipeline, fragment processing works alongside terminology mapping, validation rules, transformation logic, conflict handling, lineage tracking, and quality checks. Together, these components help turn raw FHIR exports into analytics-ready OMOP datasets for cohort analysis, outcomes research, quality reporting, population health, and AI use cases.
Read the broader guide on FHIR to OMOP conversion for healthcare analytics to see how fragment processing fits into the full mapping, normalization, and healthcare analytics workflow.








BLOGS
NEWSROOM
CASE STUDIES
WEBINARS
PODCASTS
ASSET HUB
EVENT CALENDAR 


















