Senior Databricks Data QA Engineer/Data Engineering

Title:Data Engineering

mployment Type: Three-month contract-to-hire; full-time conversion expected after three months

Position Summary

We are seeking a Senior Databricks Data QA Lead with extensive data engineering experience to lead data quality assurance and automation across healthcare payer data platforms. This role will build automated validation frameworks, establish deployment quality gates, and ensure the accuracy, reliability, performance, and compliance of enterprise data pipelines.

Key Responsibilities

• Design and implement test plans, test cases, and automated validation for ETL/ELT pipelines.
• Validate healthcare payer datasets, including claims, membership, provider, and clinical data.
• Conduct FHIR-based API testing to support CMS interoperability compliance.
• Validate HEDIS measure calculations and quality reporting data.
• Build and maintain data quality automation frameworks using Python, PySpark, and Spark SQL.
• Develop Azure DevOps test plans and QA gates for Databricks Jobs, Delta Live Tables, and Azure Data Factory deployments.
• Collaborate with product owners, data engineers, governance teams, and business stakeholders to identify and resolve data quality issues.
• Design automated load and stress tests for large-scale data pipelines.
• Ensure data QA practices adhere to HIPAA, CMS, PII protection, and payer industry requirements.
• Document QA processes, test results, and audit-ready evidence.
• Mentor and guide junior QA engineers.

Required Qualifications

• 12+ years of Data Engineering experience.
• 5+ years working exclusively on Databricks.
• Strong expertise in Azure Databricks, including Workflows, Delta Lake, Delta Live Tables, and Unity Catalog.
• Hands-on experience with Azure Data Factory pipelines, monitoring, and CI/CD.
• Proficiency in Python, PySpark, and Spark SQL for data quality automation.
• Experience validating healthcare payer data and working with FHIR APIs and HEDIS reporting.
• Experience with Azure DevOps, automated deployment pipelines, and QA gates.
• Knowledge of data governance, PII compliance, and data quality frameworks.
• Experience testing the performance and scalability of data pipelines.
• Strong communication, problem-solving, and technical leadership skills.
• Willingness to convert to full-time employment after the initial contract period.

Preferred Qualifications

• Experience with Databricks DQX.
• Experience with Databricks Asset Bundles (DAB).