Data Engineer with Databricks

Job Summary
We are seeking a Senior Data Engineer with 8+ years of experience to maintain, support, and enhance production data pipelines on Databricks. This role combines day-to-day production support — monitoring jobs, debugging failures, performance tuning — with core data engineering work such as designing scalable ETL/ELT processes, building data models, and integrating new data sources. You’ll work directly with business and technical stakeholders, so strong communication and client-facing skills are essential.

Key Responsibilities

• Own production support for Databricks Jobs/Workflows, Delta Live Tables, and scheduled notebooks; monitor runs, triage failures, perform root-cause analysis, and implement permanent fixes
• Design, develop, and maintain ETL/ELT pipelines using PySpark, Spark SQL, and Python to ingest, transform, and deliver data from diverse sources (databases, APIs, flat files, SFTP, cloud storage)
• Debug complex data issues across Spark, Delta Lake, SQL, and upstream systems; manage reprocessing, backfills, and data reconciliation
• Optimize pipeline performance and cost: cluster configuration, partitioning, Z-ordering, caching, query tuning, and job scheduling
• Write and maintain Unix/Linux shell scripts for file handling, job automation, and operational utilities
• Develop and tune complex SQL for data transformation and validation; build and maintain data models (star/snowflake, Delta Lakehouse bronze/silver/gold layers)
• Manage Unity Catalog objects, access controls, and data lineage; support schema evolution and governance standards
• Implement data-quality checks, logging, monitoring, and alerting; create runbooks and documentation for recurring operational scenarios
• Support CI/CD and release processes using Git, Databricks Repos, Databricks Asset Bundles, and/or Terraform
• Partner with analysts, data scientists, and client stakeholders to gather requirements, communicate issue status and ETAs, and deliver enhancements
• Mentor junior engineers and contribute to engineering standards and best practices
• Participate in on-call rotation for critical pipelines as needed

Required Qualifications

• 8+ years of data engineering experience in enterprise environments, including 3+ years hands-on with Databricks
• Expert-level PySpark and SQL; deep understanding of Spark internals, partitioning, shuffles, and performance tuning
• Strong Python programming skills for data processing, automation, and utilities
• Proven experience supporting production pipelines: orchestration, failure handling, retries, backfills, and incident resolution with SLAs
• Solid Unix/Linux skills including shell scripting (bash) and command-line troubleshooting
• Hands-on experience with Delta Lake, Unity Catalog, Databricks Workflows, and Auto Loader
• Experience with at least one major cloud (Azure, AWS, or Google Cloud Platform) and its storage/compute services
• Proficiency with Git and CI/CD practices
• Strong client-facing communication skills; able to work independently in a remote environment

Preferred Qualifications

• Databricks Certified Data Engineer (Associate or Professional)
• Healthcare data experience (claims, clinical, EHR, HL7/FHIR, or HR/workforce data)
• Experience with dbt, Airflow, Azure Data Factory, or similar orchestration tools
• Infrastructure-as-code experience (Terraform, Databricks Asset Bundles)
• Familiarity with data-quality frameworks (Great Expectations or similar)