Sr. Data Engineer(PA locals only )

Job Title: Sr. Data Engineer

Location: Hybrid-2 days a week onsite in downtown Philadelphia, PA- locals only within one drive distance 

Duration: 6 month contract to hire

Visa: only 

Exp Level: 10+ years

Must have: 

These roles are focused on building real time streaming data solutions that enable data-driven decisions on live devices and telemetry data. 
While the Senior Data Engineer will support batch data processes, the primary focus is on real-time streaming use cases, data services for APIs and microservices, and pipelines that feed both operational applications and ML/AI model development in a medical device IoT environment. 

Role Description :
As a Senior Data Engineer, you will play a key role in driving the design and architecture of data pipelines that consume real-time streaming data from connected medical devices. You will collaborate cross-functionally with business analysts, software engineers, machine learning engineers, and business users to implement technical approaches and infrastructure that support data consumption across the organization. 

This is a great opportunity for someone who is passionate about data engineering, thrives in environments where you can take ownership and recommend solutions, and is a self-starter who is open to learning new tools as needed. 

Qualifications:

Required Experience
•    8+ years of experience designing, building, and operating big data and real-time streaming pipelines across cloud and on-premises environments
•    5+ years applying DevOps and CI/CD practices to data and analytics workloads
•    Production experience building data services, APIs, or microservices for downstream data consumption

Required Technical Skills:
•    Strong, hands-on experience designing and building production-grade data pipelines on Databricks
•    Demonstrated experience with Spark optimization and tuning in Databricks, including performance analysis, partitioning strategies, caching, shuffle optimization, and cost-aware pipeline design
•    Strong experience with Azure cloud services for data engineering and streaming workloads
•    Experience building data services, APIs, or microservices for data consumption.
•    Data quality, validation, and privacy-aware handling for regulated or sensitive data

Preferred Technical Skills:
•    Azure Event Hubs and Azure Stream Analytics for IoT and telemetry ingestion and stream processing
•    Additional Azure data and integration services supporting batch and streaming architectures
•    Collaboration with ML teams: feature pipelines, training data, or streaming data for model development

Key Responsibilities:
•    Design and build batch and streaming data pipelines on Azure and Databricks, with primary focus on real-time IoT and telemetry use cases within a Medallion architecture.
•    Develop ETL/ELT workflows to ingest, transform, and validate large volumes of structured and unstructured data.
•    Build and maintain data services, APIs, and microservices for application, analytics, and ML/AI teams.
•    Implement real-time streaming solutions using Azure Event Hubs, Azure Stream Analytics, and related Azure integration patterns, with cost-effective throughput, partitioning, and downstream delivery to Databricks.
•    Optimize production Databricks pipelines using PySpark, Spark SQL, and Delta Lake, including Spark tuning for performance, reliability, and cost.
•    Troubleshoot and resolve complex pipeline issues across Databricks, Azure, and on-premises systems, including root cause analysis and corrective action.
•    Partner with data analysts, software engineers, ML engineers, and business stakeholders to translate requirements into technical designs and delivery priorities.
•    Apply data quality, validation, and privacy-first practices, and deliver reliable pipelines through software engineering standards, documentation, testing, and CI/CD.

Outcomes
•    Onboard to the Azure Databricks environment and contribute to troubleshooting, stabilization, and optimization of existing batch and streaming data pipelines.
•    Stand up Azure streaming ingestion for telemetry data and deliver production-ready pipelines integrated with Databricks to support API, ML, and downstream analytics consumption within the Medallion architecture.
•    Design and deliver data services or consumption patterns that enable business and ML teams to access near-real-time telemetry data reliably, securely, and on a scale.

Work Hours and Travel Requirements:

Candidates must be open to assisting in troubleshooting and analysis in the event of off-hours production problems, as needed. The IT Team works in a hybrid environment that requires a minimum of two days per week in their downtown Philadelphia office.