Back to Jobs
Recently postedRemoteOther

Lead Databricks Platform & Unity Catalog

eSolutionsFirst, LLCPhiladelphia, PA🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeRemote
LocationPhiladelphia, PA, United States
OnboardingGitComplianceDatabricksGitHub ActionsTerraformUnityVaultAzureCI/CDAzure DevOps

Job description

Lead Databricks Platform & Unity Catalog (Azure)

Job Type: Contract (6-12 Months)

Philadelphia, PA / Vienna, VA - Hybrid / Open to remote job

Job description:

Job summary

  • We are seeking a platform led to stand-up Azure Databricks and Unity Catalog for a regulated financial services client.
  • You will design the workspace and catalog structure, implement the client's access and data-protection policies, and lead a small team of platform engineers.
  • The client's governance and security teams set policy; you turn it into a working, auditable platform.

Key responsibilities

  • Design the workspace, environment and catalog structure for development, test and production.
  • Implement the Unity Catalog access model: groups, grants, row filters, column masks and tag-based policies.
  • Integrate identity with Microsoft Entra ID, including provisioning and service principals.
  • Configure external locations, storage credentials and network controls with the client's cloud team.
  • Establish lineage, audit logging and access-review evidence for security and compliance.
  • Define the CI/CD and infrastructure-as-code standards for the platform and enforce separation of duties.
  • Write architecture decision records and take them through the client's architecture and security review.
  • Lead and mentor platform engineers; hand over runbooks to the client's run team.
  • Required qualifications
  • 10+ years in data platform engineering, including 3+ years on Databricks and 2+ years implementing Unity Catalog in production.
  • Hands-on Azure experience: Entra ID, ADLS Gen2, networking, Key Vault.
  • Fine-grained access control in Unity Catalog, including row filters and column masks on sensitive data.
  • Infrastructure as code with Terraform, and CI/CD on Azure DevOps or GitHub Actions.
  • Experience presenting designs to enterprise architecture and information security reviewers.
  • Experience leading a technical team of three or more.

Preferred qualifications

  • Financial services experience, including controls for client and account data.
  • A Unity Catalog migration or new platform build for a regulated client.
  • Lakehouse Federation, Delta Sharing and Databricks system tables.
  • Databricks Data Engineer Professional or platform architect certification.
  • Conditions of engagement
  • Must be based in, and work from, the United States for the full engagement.
  • Work is performed in client-provisioned environments on client-managed or approved devices.
  • Background check and client onboarding requirements must be completed before start.
  • AI coding and analysis tools may be used only on endpoints the client has approved.
  • Experience in financial services, wealth management or another regulated industry is strongly preferred.

Will your resume get past this employer's screening? Check your resume against this job, free

Similar jobs