Quick overview
Job description
Role Summary
Design, develop, and maintain scalable data pipelines and data assets on Databricks and the Microsoft Azure Data Platform to support Business Intelligence and Advanced Analytics objectives.
- Lead the technical migration of legacy databases and on-premises data systems to a cloud-based lakehouse architecture using Delta Lake, including data modeling, transformation logic, and post-migration data validation.
Role and Responsibilities:
- Design, develop, and maintain scalable data pipelines and data assets on Databricks and the Microsoft Azure Data Platform to support Business Intelligence and Advanced Analytics objectives.
- Lead the technical migration of legacy databases and on-premises data systems to a cloud-based lakehouse architecture using Delta Lake, including data modeling, transformation logic, and post-migration data validation.
- Develop distributed data processing applications using PySpark, Scala Spark, Python, and SQL for batch, incremental(CDC),and real-time streaming workloads.
- Build and customize data integration tools, data warehouses, data marts, and data lakes using Azure Data Factory, ADLS Gen2, and Kafka.
- Analyze business and system requirements and translate them into technical designs, data models, and software specifications aligned with enterprise architecture.
- Implement data governance, security, and data quality controls, including access management, lineage tracking, and automated data validation frameworks.
- Develop CI/CD pipelines and infrastructure-as-code for automated testing and deployment of data solutions using Databricks Asset Bundles and Git-based version control.
- Optimize performance, reliability, and cost of Spark workloads and cloud infrastructure through job tuning, cluster configuration, and storage optimization.
- Create tools and automation that help client engineering teams derive value from their data.
- Perform hands-on development, lead code reviews and testing, and establish engineering best practices across the team.
- Review solution approaches and provide technical guidance to onshore and offshore engineers to ensure delivery quality.
- Collaborate with client stakeholders to understand data needs and support timely delivery of data platform initiatives
Requirement/Competencies:
Bachelor’s or Master’s (preferred) degree in in a quantitative or technical field such as Computer Science, Information Technology, Computer Engineering or equivalent
8+ years of experience comprising development for data systems at scale (building data warehouse, data marts, data lakes, ETL, analytic solutions, etc.)
Experience leading data migration / cloud transformation engagements
Experience with and understanding of Databricks, Azure Data Platform is a must.
Haystack's AI agent can fill in and submit this application for you. How AI auto apply works
Will your resume get past this employer's screening? Check your resume against this job, free
Similar jobs
- GA

Senior Databricks Apps Full-Stack Engineer Auto Apply
Growth Acceleration Partners · Austin · Hybrid · Posted 3 Nov 2026 🇺🇸
FastAPISQLApache+7TechnologySalary not listed - GA

Staff Data EngineerRecently posted Auto Apply
Gamechanger · GameChanger Remote - US · Remote · Posted 18 Nov 2026 🇺🇸
SQLAWSMLOps+8Technology$170k - $185k/yr - GH

Staff Data EngineerRecently posted Auto Apply
Garner Health · New York City · Hybrid · Posted 18 Nov 2026 🇺🇸
SQLAWSLooker+13Technology$256k - $287k/yr - LA
Data Engineer IIRecently posted Auto Apply
Lantern · Dallas · Hybrid · Posted 19 Nov 2026 🇺🇸
Talent AcquisitionGCPSQL+7TechnologySalary not listed - CL

Founding Data EngineerRecently posted Auto Apply
Clera · San Francisco · On-site · Posted 18 Nov 2026 🇺🇸
SQLBigQueryERP+5Technology$170k - $200k/yr - MI
Lead Forward Deployed Engineer, Data Auto Apply
Midstream · United States · Remote · Posted 12 Nov 2026 🇺🇸
OnboardingWorkdayOracle+9EngineeringSalary not listed - MA

Senior Data EngineerRecently posted Auto Apply
Machinify · United States · Remote · Posted 18 Nov 2026 🇺🇸
SQLApache SparkAWS+3Technology$180k - $220k/yr - MA

Staff Data Engineer, Core MigrationsRecently posted Auto Apply
Machinify · United States · Remote · Posted 18 Nov 2026 🇺🇸
SQLT-SQLApache Spark+7Technology$230k - $260k/yr - MA

Senior Data Engineer - AnalyticsRecently posted Auto Apply
Machinify · United States · Remote · Posted 18 Nov 2026 🇺🇸
SQLMicrosoft ExcelGit+14Technology$170k - $200k/yr - AB

Associate Data Engineer IIRecently posted Auto Apply
AbbVie · Mettawa, IL · Hybrid · Posted 18 Nov 2026 🇺🇸
Data AnalysisTechnology$65.5k - $125.5k/yr - ME

Customer Data Platform (CDP) ManagerRecently posted Auto Apply
Monster Energy · USA - Corona · Hybrid · Posted 18 Nov 2026 🇺🇸
SQLLookerCRM+3$116.3k/yr - CL

DEXPI Specialist (P&ID Data Exchange)Recently posted Auto Apply
Clera · United States · On-site · Posted 18 Nov 2026 🇺🇸
$90/hr
