Lead Data Engineer with Databricks With Insurance Domain

JOB DESCRIPTION

POSITION: Lead Data Engineer with Databricks  With Insurance Domain
DURATION: Long term
Location: Charlotte, NC - Onsite

Roles & Responsibilities

• Develop high-quality, scalable ETL/ELT pipelines using Databricks technologies including Delta Lake, Auto Loader etc.,
• Excellent programming and debugging skills in Python
• Strong hands-on experience with PySpark to build efficient data transformation and validation logic
• Must be proficient in at least one cloud platform: AWS, Google Cloud Platform, or Azure
• Need someone with Insurance domain experience - property and casualty
• Implement ingestion patterns using Auto Loader with checkpointing and schema evolution for structured and semi-structured data
• Build secure and observable DLT pipelines with DLT Expectations, supporting Bronze/Silver/Gold medallion layering
• Configure Unity Catalog: set up catalogs, schemas, user/group access, enable audit logging, & define masking for PII fields
• Enable secure data access across domains & workspaces via Unity Catalog External Locations, Volumes, & lineage tracking
• Access and utilize data assets from the Databricks Marketplace to support enrichment, model training, or benchmarking
• Collaborate with data sharing stakeholders to implement Delta Sharing — both internally and externally