Databricks Architect

Job Title: Databricks Architect

Location: Los Angeles, CA or New York, NY (Onsite)

Overview

We are seeking an experienced Databricks Architect to design and implement scalable data solutions on the Databricks Lakehouse Platform. The ideal candidate will have strong expertise in Databricks, Spark, Delta Lake, cloud platforms (Azure/AWS/Google Cloud Platform), and modern data engineering practices. This role involves architecting enterprise data platforms, optimizing performance, and enabling analytics and AI/ML workloads.

Key Responsibilities

• Design end-to-end Databricks Lakehouse architectures.
• Build scalable ETL/ELT pipelines using PySpark, Spark SQL, and Databricks Workflows.
• Implement Delta Lake (Bronze/Silver/Gold), Structured Streaming, and Auto Loader.
• Optimize Spark jobs, clusters, and platform performance.
• Integrate Databricks with Azure, AWS, or Google Cloud Platform services.
• Implement CI/CD using Git, Databricks Repos, and Terraform.
• Ensure data governance, security, and compliance using Unity Catalog.
• Collaborate with Data Engineers, Data Scientists, and business stakeholders while providing architectural guidance.

Required Skills

• 7+ years of Data Engineering/Architecture experience.
• 3+ years of hands-on Databricks experience.
• Strong expertise in Databricks, Spark, PySpark, SQL, and Delta Lake.
• Experience with Azure (ADF, ADLS, Azure DevOps), AWS (S3, Glue, Lambda), or Google Cloud Platform (GCS, Dataflow, Pub/Sub).
• Experience with CI/CD, Git, Terraform, and Databricks Workflows.
• Strong understanding of data governance, performance tuning, and cloud architecture.

Preferred

• Databricks Certifications.
• Experience with MLflow, Feature Store, or MLOps.
• Experience in BFSI, Healthcare, or other regulated industries.