Databricks Engineer Chicago - IL - Illinois

Senior Databricks Engineer

Job Summary:

Seeking a Senior Databricks Engineer with strong experience in Databricks, Apache Spark, Delta Lake, and cloud data engineering. The candidate will design, develop, optimize, and maintain scalable data pipelines supporting analytics and AI-driven healthcare applications.

Responsibilities:

Design and develop scalable data engineering solutions using Databricks and Apache Spark.

Build and optimize ETLELT pipelines using PySpark, Python, or Scala.

Implement Delta Lake architecture and distributed data processing solutions.

Optimize Spark jobs, Databricks clusters, pipelines, performance, and cloud costs.

Develop CICD, testing, monitoring, and deployment processes for Databricks workloads.

Troubleshoot complex data pipeline and Spark performance issues and perform root-cause analysis.

Support ML model deployment and lifecycle management within Databricks.

Collaborate with data engineers, data scientists, architects, and business stakeholders in Agile environments.

Required Skills:

Databricks

Apache Spark PySpark

Delta Lake

Python or Scala

ETLELT & Data Engineering

AzureAWSGoogle Cloud Platform

Preferred: Azure Databricks, MLflow, Spark SQL, CICD, Azure DevOps, Terraform, Machine Learning, Data Governance, Healthcare Analytics.

Education: Bachelors degree in Computer Science, Engineering, IT, or related field preferred.