Databricks Architect / Resident Solution Architect

Job Title: Databricks Architect / Resident Solution Architect
Openings: 2
Location: San Francisco, CA – Local preferred; Remote considered
Work Hours: Must be able to work PST hours
Duration: 8–12 Months
Visa: Any
Rate: flexible
Industry: Banking / Financial Services
Experience: 12–15+ Years

Job Description

We are looking for a highly experienced Databricks Architect / Resident Solution Architect with strong hands-on Data Engineering and Databricks architecture experience.

The ideal candidate should have extensive experience designing and delivering enterprise-scale Databricks Lakehouse solutions and must have prior Banking/Financial Services domain experience.

Key Responsibilities

• Architect and deliver enterprise-scale solutions using the Databricks Lakehouse Platform.

• Lead end-to-end Databricks implementations from architecture and design through production deployment.

• Design scalable, secure, high-performance, and cost-efficient data platforms.

• Develop and optimize ETL/ELT pipelines using Databricks, Apache Spark, Delta Lake, and Structured Streaming.

• Design modern data warehouse and Lakehouse architecture solutions.

• Implement data governance and security using Unity Catalog.

• Optimize Spark workloads, including partitioning, execution plans, memory management, and runtime performance.

• Design cloud-native data solutions across AWS, Azure, and/or Google Cloud Platform.

• Implement CI/CD and DevOps practices for enterprise data platforms.

• Work with MLflow and MLOps practices for machine learning lifecycle management, model deployment, and monitoring.

• Troubleshoot complex Databricks, Spark, and distributed data-processing issues.

• Provide architecture recommendations based on business and technical requirements.

• Work directly with client stakeholders and provide technical leadership throughout implementation.

Required Skills

• 12–15+ years of experience in Data Engineering, Data Platforms, Data Analytics, and Modern Data Warehouse solutions.

• 10+ years of consulting and client-facing delivery experience.

• Banking/Financial Services experience is mandatory.

• Experience delivering 6–8+ end-to-end Databricks implementations as a hands-on developer, technical lead, or solution architect.

• Advanced hands-on experience with Databricks and Apache Spark.

• Strong experience with Delta Lake, Structured Streaming, Unity Catalog, Workflows, MLflow, and Delta Live Tables (DLT).

• Strong Spark performance tuning and optimization experience.

• Deep hands-on experience with at least one major cloud platform: AWS, Azure, or Google Cloud Platform.

• Experience developing large-scale ETL/ELT data pipelines.

• Experience with DevOps and CI/CD using Azure DevOps, GitHub Actions, GitLab CI/CD, or Jenkins.

• Working knowledge of MLOps, model deployment, lifecycle management, and monitoring.

• Strong understanding of distributed data processing and scalable data architecture.

• Strong client-facing communication and consulting skills.

Certification

Required/Strongly Preferred: Databricks Data Engineer Professional certification or equivalent advanced Databricks certification.

Good to Have: Databricks Solutions Architect Champion recognition/program experience.

Please share your updated resume along with your current visa/work authorization status at for further consideration.