Senior Databricks Platform Engineer

Senior Databricks Platform Engineer

Eagan, MN – Local Candidates Only

Contract

Direct Client 

Skills:

Databricks
Platform Governance & Cost Management 
Architecture Design for Data Pipelines 
Security & Networking Expertise

Key Responsibilities

1.Databricks Platform Governance & Cost Management

Establish and maintain Databricks platform governance, access controls, and operational standards.

Monitor and optimize Databricks resource utilization, compute workloads, and cloud spending.

Implement cost-control strategies, cluster policies, and workload optimization practices.

Track platform usage and identify opportunities to improve performance and reduce costs.

Support data security, compliance, and governance requirements.

2. Architecture Design for Data Pipelines

Design and implement scalable, reliable, and efficient data pipeline architectures using Databricks.

Develop and optimize ETL/ELT workflows for batch and streaming data processing.

Work with Delta Lake, Apache Spark, PySpark, and related data engineering technologies.

Design data ingestion, transformation, storage, and integration solutions.

Troubleshoot pipeline performance issues and improve data processing efficiency.

Collaborate with data engineers, architects, and business teams to deliver data platform solutions.

3. Security & Networking Expertise

Implement secure access management, authentication, authorization, and role-based access controls.

Configure and manage secure connectivity between Databricks and cloud resources.

Support networking configurations, including private connectivity, network security rules, and firewall requirements.

Apply security best practices for data access, secrets management, encryption, and sensitive information protection.

Work with cloud infrastructure teams to maintain a secure and compliant data platform.

Required Skills

Strong hands-on experience with Databricks platform administration and architecture.

Experience with Databricks governance, access management, and cost optimization.

Expertise in designing scalable data pipelines and ETL/ELT architectures.

Strong knowledge of Apache Spark, PySpark, and Delta Lake.

Experience with cloud security, networking, and private connectivity.

Knowledge of cloud infrastructure on AWS, Azure, or Google Cloud Platform, preferably aligned with the client's environment.

Strong troubleshooting, performance tuning, and problem-solving skills.

Excellent communication and collaboration skills.

Preferred Qualifications

Experience with Unity Catalog and centralized data governance.

Knowledge of Databricks cluster policies, compute management, and workload monitoring.

Experience with CI/CD, infrastructure as code, and automated platform deployment.

Familiarity with enterprise data security, compliance, and audit requirements.

Relevant Databricks or cloud platform certifications are a plus.