Quick overview
SeniorityMid Senior
Work modeHybrid
LocationUnited States
SQLApache SparkAWSApacheDatabricksKafkaPythonQuota ManagementRabbitMQReconciliationUnityAzureAzure Data FactoryDelta Lake
Job description
Focus: Scalability, Performance Optimization & Data Integration
Experience: 8+ years in Data Engineering, including 3+ years of hands-on Azure Databricks experience.
Role Overview
We are looking for a highly experienced, hands-on Azure Databricks Data Engineer to build and optimize scalable, high-performance data pipelines for a large-scale healthcare analytics platform.
The engineer will integrate an existing Python-based data processing and enrichment engine with Azure Databricks, reuse existing transformations, and enable bidirectional data movement between the engine, Azure Data Lake, and Microsoft Fabric.
The primary focus is scalability, performance optimization, near-real-time processing, and efficient Azure resource utilization.
Key Responsibilities
- Data Lake Integration: Build and optimize data pipelines between existing Python processing services, Azure Databricks, ADLS Gen2, and Microsoft Fabric using Bronze, Silver, and Gold architecture.
- Spark Performance Optimization: Optimize PySpark and Spark SQL workloads involving billions of records, complex joins, large-table MERGE operations, data skew, shuffles, and unnecessary table scans.
- Scalability & Resource Management: Design scalable multi-tenant pipelines with efficient cluster sizing, autoscaling, parallel processing, workload orchestration, and Azure compute quota management.
- Streaming & Incremental Processing: Implement high-performance CDC, micro-batch, and incremental pipelines using Structured Streaming, Event Hubs, and Azure Data Factory. Support updates, deletes, retries, and replay.
- Transformation Reuse: Integrate existing Python and SQL-based business transformations, ensuring data consistency, correctness, and minimal redevelopment.
- Data Integration & Quality: Enable reliable bidirectional data flows between the existing processing engine and the lakehouse, with automated reconciliation, schema evolution, monitoring, and failure recovery.
- Fabric Integration: Publish optimized and curated datasets to Microsoft Fabric for downstream analytics and reporting.
Required Technical Skills
- Strong hands-on expertise in Azure Databricks, Apache Spark, PySpark, and advanced SQL.
- Deep understanding of Spark internals, query execution plans, partitioning, clustering, joins, shuffles, memory optimization, and performance tuning.
- Experience with Delta Lake, ADLS Gen2, Azure Data Factory, Azure Event Hubs, and Structured Streaming.
- Proven experience designing multi-terabyte, multi-tenant, and high-volume data pipelines.
- Strong knowledge of CDC, incremental processing, schema evolution, data reconciliation, and distributed processing.
- Experience optimizing Databricks clusters, job orchestration, concurrency, and Azure compute costs.
- Strong Python development and production troubleshooting skills.
Preferred Skills
- Microsoft Fabric, OneLake, Apache Iceberg, and Unity Catalog.
- Kafka, RabbitMQ, Debezium, and asynchronous data integration.
- AWS-to-Azure cross-cloud integration.
- Healthcare data processing and XML/JSON transformation experience.
Ideal Candidate
A strong, hands-on data engineer who has successfully optimized production-scale Databricks pipelines and can independently diagnose performance bottlenecks, improve processing speed, and design scalable, cost-efficient solutions.
Must be able to demonstrate real-world Spark/Databricks performance improvements with measurable results.
Similar jobs
- AF
Engagement Manager - Fintech/NYCRecently posted Auto Apply
Adaptive Financial Consulting · New York · Hybrid 🇺🇸
Diversity & InclusionMental HealthZoom$200k - $250k/yr - TH
Data Engineering ManagerRecently posted Auto Apply
Trella Health · Remote - Atlanta · Remote 🇺🇸
SQLAWSETL+16Salary not listed - TE
Telecom Managers: Corporate Mobile Device ProgramsRecently posted Auto Apply
Terac · United States · Remote 🇺🇸
Fleet ManagementVendor ManagementReact+1Salary not listed - TA
Director of ProductRecently posted Auto Apply
TailorCare · United States · Hybrid 🇺🇸
401kSalary not listed - SP
Carrier Account CoordinatorRecently posted Auto Apply
Spot Inc. · Charlotte · Hybrid 🇺🇸
Microsoft OfficeAuditingSalary not listed - SP
Senior Pricing AnalystRecently posted Auto Apply
Spot Inc. · Chicago · Hybrid 🇺🇸
SQLMicrosoft ExcelMicrosoft Office+4Salary not listed - SP
Pricing AnalystRecently posted Auto Apply
Spot Inc. · Chicago · Hybrid 🇺🇸
SQLMicrosoft ExcelMicrosoft Office+4Salary not listed - RZ
Procesador(a) de Pollo – Primer TurnoRecently posted Auto Apply
R.W. Zant · Los Angeles · Hybrid 🇺🇸
HACCP$18/hr - RZ
Procesador(a) de PolloRecently posted Auto Apply
R.W. Zant · Los Angeles · Hybrid 🇺🇸
HACCP$19/hr - RZ
Asociado de Almacén 1st ShiftRecently posted Auto Apply
R.W. Zant · Los Angeles · Hybrid 🇺🇸
$18/hr - SW
Facilities TechnicianRecently posted Auto Apply
Swarmaero · Oxnard · On-site 🇺🇸
Microsoft ExcelMicrosoft OfficeAssembly+1Salary not listed - UL
Tree Service ClimberRecently posted Auto Apply
United Land Services · Jacksonville · On-site 🇺🇸
ComplianceSalary not listed