Data Pipeline & Ingestion Engineer – ODL Program

Job Title: Senior Data Pipeline & Ingestion Engineer
Experience: 5+ Years
Employment Type: Full-Time
Level: Mid / Senior / Lead

About the Role

We are looking for an experienced Data Pipeline & Ingestion Engineer to design and build large-scale batch and streaming data pipelines.

The role focuses on Kafka, Python/Java, SQL, ETL, data ingestion, Medallion/Lakehouse architecture, CDC, data quality, reconciliation, and entity resolution/MDM.

You will work on the core data platform responsible for ingesting data from multiple legacy and enterprise systems, transforming it into standardized data models, and ensuring data quality and reconciliation before publishing.

This reflects the uploaded ODL requirement, which specifically emphasizes Kafka, batch/stream ingestion, medallion layers, reconciliation, identity matching, source-to-canonical mappings, and data contracts.

Responsibilities

• Design and build scalable batch and streaming data pipelines.
• Develop data ingestion solutions using Kafka.
• Implement consumer/producer, replay, DLQ, and idempotent processing patterns.
• Build and maintain Bronze, Silver, and Gold Medallion data layers.
• Implement CDC, watermark, checkpoint, retry, and batch-to-stream handoff patterns.
• Develop strong data-quality validation and reconciliation processes.
• Implement quarantine, reprocessing, and data-quality scoring.
• Perform count, record-level, and financial reconciliation.
• Develop identity matching and entity resolution / MDM solutions.
• Implement record matching, deduplication, golden records, and survivorship rules.
• Build and maintain source-to-canonical data mappings and crosswalks.
• Work with structured and semi-structured enterprise datasets.
• Implement schema validation and schema evolution.
• Support production monitoring, troubleshooting, and performance optimization.

Mandatory Skills

Good to Have

• Informatica MDM
• Reltio
• Probabilistic record matching
• Golden-record implementation
• Avro / Protobuf
• Schema Registry
• Mainframe or legacy RDBMS ingestion
• Financial reconciliation
• Healthcare / Benefits domain experience