Databricks Engineer, HIPAA

Role: Databricks Engineer, HIPAA

Location of work: Remote

Duration: 12 months

Description

Overview of position:

This is a hands-on data engineering role dedicated to CPAL's HIPAA-regulated data workflows, primarily in support of our Maternal Health big bet and initiatives like Trust Her. CPAL is actively setting itself up to be able to handle Protected Health Information (PHI) and to enter into Business Associate Agreements, and this role is central to standing that up. You will help build, and then own, how PHI moves through our platform end to end: secure ingestion, storage, transformation, de-identification, and controlled sharing inside a HIPAA-compliant Databricks environment. This is not a role where compliance is someone else's job. You are the person who makes sure the pipelines, access controls, and documentation for regulated data are correct, defensible, and auditable. We run a Databricks lakehouse on a medallion architecture with Unity Catalog for governance, and we are still actively building out our standards and conventions. That is especially true for how we handle PHI, which is largely being built now, so you will help define the patterns rather than just inherit them. Day to day, the weight of the role sits in two places: building and maintaining the pipelines that carry regulated health data, and stewarding the governance and controls that keep that data compliant. We are hiring this primarily as a mid-level role, with room to grow into more ownership over time.

What you ll do:

HIPAA data pipeline engineering (core)

• Help stand up and then maintain secure batch pipelines that ingest, clean, and transform PHI from clinical and community partners into the lakehouse, following least-privilege and minimum-necessary principles throughout.
• Land raw PHI into appropriately restricted zones, then build the transformations, and where needed the de-identification or tokenization, that turn it into analysis-ready datasets for program teams.
• Implement data quality checks, validation, and audit logging so that every movement of regulated data is traceable and defensible.
• Help establish, and then support, controlled data sharing with partners under Business Associate Agreements and data-use agreements as we build out that capability, including via Delta Sharing where appropriate.
• Partner with program, compliance, and legal staff to translate data-use agreements, consent, and IRB constraints into concrete technical controls.

HIPAA platform governance (core)

• Help stand up and maintain a HIPAA-eligible Databricks environment: configure the access, encryption, and audit controls required to keep PHI workloads compliant, under the direction of the Director of Data Operations and compliance staff.
• Implement fine-grained access control in Unity Catalog, including row- and column-level security, dynamic views for masking, and tightly scoped groups for regulated catalogs and schemas.
• Build and maintain audit trails, and support periodic access reviews for PHI, keeping the documentation that underpins compliance current.
• Implement the data classification and handling standards that separate PHI, de-identified, and public data, and contribute to shaping them as they develop.
• Configure and manage tiered access for staff, contractors, and external partners who touch regulated data, translating the permissions defined in each agreement into technical controls.

You may also

• Support de-identified analytics and reporting for the maternal health program.
• Contribute to broader platform and governance work as needs arise, given that HIPAA controls and general lakehouse governance overlap heavily.

What we re looking for:

Must have

• Solid data engineering experience building and maintaining production ETL/ELT pipelines.
• Hands-on experience with Databricks, or a strong and transferable modern lakehouse /Spark background with a clear ability to ramp quickly: Delta Lake, jobs and workflows, and SQL/PySpark transformations.
• Strong SQL and Python.
• Direct experience handling PHI or other regulated and sensitive data, with a working understanding of HIPAA (the Privacy and Security Rules), what constitutes PHI, and the basics of de-identification.
• A working grasp of access control and data governance: permissions, principals and groups, and row- and column-level security; Unity Catalog experience is a strong plus.
• Comfort implementing security controls in practice: least privilege, minimum necessary, audit logging, and encryption in transit and at rest.
• Version control (Git) and collaborative development practices.
• Clear communication skills, with the ability to work alongside compliance and program staff and translate their requirements into technical controls.
• Sound judgment and discretion when handling confidential data.

Nice to have

• Experience configuring or operating a HIPAA-compliant cloud or Databricks environment (for example, Databricks' compliance security profile on AWS).
• Experience standing up HIPAA compliance in a cloud data platform, not just operating within one that was already built.
• Familiarity with de-identification approaches (Safe Harbor and Expert Determination) and with tokenization or hashing of identifiers.
• Prior work with health, clinical, or public-health data (for example claims, EHR, or vital records data).
• Familiarity with Business Associate Agreements, data-use agreements, and consent or IRB constraints.
• Unity Catalog administration and Delta Sharing experience.
• Experience with R alongside Python.
• Cloud platform familiarity (AWS preferred given our Databricks footprint; other clouds welcome).
• Workflow orchestration and CI/CD practices for data.
• Awareness of related frameworks and regulations (for example HITECH and relevant state health-privacy law).
• Background in nonprofit, public-sector, maternal and child health, or social-science data.