Data Platform Lead

Databricks | Data Engineering and Analysis | Cloud Security | Governance | Encryption | Multi-Tenant Platforms

Required Qualifications

Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, or a related field, or equivalent practical experience.

Strong experience leading the design and implementation of enterprise cloud data platforms, with substantial hands-on Databricks experience.

Strong working knowledge of Apache Spark, Delta Lake, Databricks Workflows, Unity Catalog, SQL, and Python. Scala experience is beneficial.

Demonstrated ability to perform complex data analysis, profiling, reconciliation, debugging, performance analysis, and root-cause investigation using large datasets.

Experience implementing production-grade batch, streaming, micro-batch, event, and file-based ingestion patterns, including schema evolution, replay, backfill, and idempotent processing.

Experience with cloud-native data services, object storage, IAM, private networking, key management, logging, monitoring, infrastructure as code, and CI/CD. AWS experience is preferred; Azure or Google Cloud Platform experience is also relevant.

Strong knowledge of data security and governance concepts, including least privilege, identity federation, service principals, classification, lineage, retention, masking, audit logging, DLP, controlled data sharing, and regulated or customer-sensitive data handling.

Experience designing or operating dedicated-tenant, multi-tenant, or customer-isolated platforms, including tenant lifecycle, logical and physical isolation, resource governance, and cross-tenant security testing.

Experience implementing encryption in transit and at rest, cloud KMS or HSM integrations, customer-managed keys, BYOK, key rotation, separation of duties, and cryptographic control evidence.

Strong architecture, technical documentation, stakeholder management, and engineering leadership skills, with the ability to convert ambiguous requirements into executable designs and delivery plans.

Preferred Qualifications

Experience with Databricks on AWS, including S3, KMS, PrivateLink, VPC endpoints, IAM roles, CloudTrail, CloudWatch, and enterprise network controls.

Experience with Kafka or equivalent event-streaming platforms and cloud edge or integration services.

Experience with Databricks Asset Bundles, Terraform, Git-based workflows, automated testing, release pipelines, policy as code, and environment promotion.

Experience building data-quality frameworks, lineage, observability, operational dashboards, and cost or usage reporting by tenant.

Experience with Delta Sharing, APIs, BI tools, MLflow, AI/ML workloads, or governed data-product consumption patterns.

Knowledge of Zero Trust, NIST CSF, CIS Controls, security architecture reviews, threat modeling, penetration testing, exception management, and audit evidence practices.

Experience in semiconductor manufacturing, R&D, lab, metrology, equipment telemetry, or OT-integrated data environments is an advantage.

Relevant Databricks, cloud architecture, data engineering, security, or governance certifications are desirable.