Databricks Resident Solution Architect

Location: San Francisco, CA
Work Arrangement: Remote, PST Hours
Employment Type: Contract
Duration: 12 months

Position Overview

We are seeking a Databricks Resident Solution Architect with extensive experience in data engineering, data platforms, analytics, and modern data warehouse solutions. This client-facing role will lead Databricks implementations, provide architecture guidance, and work directly with clients to design and deliver scalable cloud-native data platforms and Databricks solutions.

Key Responsibilities

• Lead end-to-end Databricks implementations as a hands-on developer, technical lead, or solution architect.
• Serve as a client-facing technical advisor, understanding business requirements and recommending appropriate Databricks solutions.
• Design and implement cloud-native data platforms across AWS, Azure, and/or GCP.
• Architect scalable ETL/ELT pipelines using Databricks, Delta Lake, Structured Streaming, and modern data engineering frameworks.
• Design and optimize large-scale distributed data processing workloads for performance, scalability, and cost efficiency.
• Apply advanced Apache Spark expertise, including partitioning, execution plans, memory management, and performance optimization.
• Implement DevOps and CI/CD practices using Azure DevOps, GitHub Actions, GitLab CI/CD, Jenkins, or similar tools.
• Apply MLOps principles for machine learning lifecycle management, model deployment, and monitoring.
• Leverage Databricks Lakehouse capabilities including Delta Lake, Unity Catalog, Workflows, MLflow, and Delta Live Tables.
• Troubleshoot complex data platform challenges and recommend architecture patterns aligned with business and technical requirements.
• Support client discussions and demonstrate the value and capabilities of Databricks solutions.

Required Qualifications

• 12–15+ years of experience in data engineering, data platforms, data analytics, and modern data warehouse solutions.
• 10+ years of consulting and client-facing delivery experience.
• Demonstrated experience delivering 6–8+ end-to-end Databricks implementations.
• Advanced Databricks certification, such as Databricks Data Engineering Professional certification, with completion of relevant learning paths and coursework.
• Strong expertise designing and implementing cloud-native data platforms across AWS, Azure, and/or GCP, with deep hands-on expertise in at least one cloud platform.
• Advanced knowledge of Apache Spark, including performance optimization, partitioning strategies, execution plans, memory management, and runtime internals.
• Extensive hands-on experience with Databricks, Delta Lake, Structured Streaming, and scalable ETL/ELT pipelines.
• Experience implementing production-grade DevOps and CI/CD for data solutions.
• Working knowledge of MLOps, machine learning lifecycle management, model deployment, and monitoring.
• Strong knowledge of the Databricks Lakehouse Platform and its core capabilities.
• Experience tuning large-scale distributed workloads and designing highly performant, scalable, and cost-efficient solutions.
• Strong troubleshooting and solution architecture skills.

Preferred Qualifications

• Databricks Solutions Architect Champion program participation or equivalent advanced Databricks expertise.
• Experience working with public sector organizations