Senior Databricks Platform Engineer (Platform Administrator)

Duration: Approximately 5 Months
Location: Dallas, TX
Work Arrangement: Onsite availability required (please confirm)

Role Summary

We are seeking a Senior Databricks Platform Engineer to design, implement, and manage scalable, enterprise-grade data platforms using Databricks. The ideal candidate will have strong hands-on experience with data engineering, cloud platforms, platform administration, solution architecture, and stakeholder engagement.

Required Skills & Experience

• Databricks: Clusters, notebooks, Delta Lake, MLflow, Unity Catalog, workspace administration, and governance.

• Cloud Platforms: Hands-on experience with AWS, Azure, or Google Cloud Platform.

• Data Engineering: Apache Spark, Python, SQL, and distributed data processing.

• Solution Architecture: End-to-end data architecture, ingestion, transformation, storage, and analytics.

• Streaming Technologies: Structured Streaming, Kafka, AWS Kinesis, or Azure Event Hubs.

• DevOps: CI/CD pipelines using Azure DevOps, GitHub Actions, Jenkins, or similar tools.

• Platform Administration: Cluster configuration, performance tuning, access control, security, monitoring, and cost optimization.

• Leadership: Technical design discussions, code reviews, mentoring, and stakeholder collaboration.

• Methodologies: Agile development and delivery practices.

Preferred Qualifications

• Databricks Data Engineer Professional or relevant architecture certification.

• AWS, Azure, or Google Cloud Platform cloud certifications.

• Experience with Tableau, Power BI, or Looker.

• Knowledge of machine learning workflows and MLOps.

• Experience with enterprise data modernization and cloud migration projects.

Key Responsibilities & Deliverables

1. Solution Architecture & Design

• Design end-to-end architectures using the Databricks Lakehouse Platform.

• Architect scalable ETL/ELT pipelines, streaming solutions, and analytics platforms.

• Define data models, storage strategies, and integration patterns.

• Optimize cluster configurations, platform performance, and resource utilization.

2. Technical Leadership

• Lead architecture discussions and technical design workshops.

• Establish engineering best practices and reusable components.

• Conduct code reviews and mentor data engineers.

3. Stakeholder Engagement

• Translate business requirements into scalable technical solutions.

• Prepare architecture diagrams, solution proposals, and implementation plans.

• Collaborate with engineering, product, and analytics teams.

4. Data Governance, Security & Compliance

• Implement Unity Catalog, data lineage, access controls, and workspace governance.

• Establish monitoring, auditing, and data quality practices.

• Align platform implementations with applicable security and compliance requirements.

5. Continuous Improvement

• Evaluate new Databricks features and complementary technologies.

• Identify opportunities for modernization, performance improvement, and cost optimization.

• Improve data platform reliability, scalability, and operational efficiency.