Senior Data Engineer – Databricks / PySpark / dbt

Required Skills

• 8+ years of hands-on Data Engineering experience.

• Strong production experience with Databricks and PySpark.

• Strong Python and SQL programming skills.

• Hands-on experience with dbt for data transformation and modeling.

• Strong implementation experience with Unity Catalog, including governance, access control, catalogs, schemas, storage credentials, and external locations.

• Strong understanding of Delta Lake and Lakehouse architecture.

• Experience with Bronze/Silver/Gold (Medallion) architecture, including understanding of its limitations and alternative design approaches.

• Strong understanding of Databricks Serverless vs. Classic Compute and the appropriate use cases/trade-offs.

• Hands-on experience designing on-premise to Databricks data pipelines, including incremental/CDC processing.

• Knowledge of modern Databricks capabilities, including Lakeflow and Lakebase.

• Experience with data quality, monitoring, error handling, performance optimization, and production troubleshooting.

• Experience with CI/CD, Git, and infrastructure/deployment practices.

Healthcare / Pharma Experience

• Healthcare or Pharma domain experience strongly preferred.

• Experience working with large-scale healthcare, clinical, claims, patient, provider, pharmaceutical, or other regulated datasets is a plus.

• Understanding of data governance, security, privacy, and controlled access is preferred.

AI / GenAI & Modern Engineering

• Exposure to Databricks AI/GenAI capabilities and modern AI-enabled data engineering.

• Understanding of AI coding assistants and coding agents, including how agents can support development, testing, debugging, documentation, and refactoring.

• Experience working with AI/LLM-enabled data platforms is a plus.