Databricks Architect

Job Summary

We are seeking an experienced Databricks Architect to design and implement scalable, secure, and high-performance data platforms using Databricks and cloud technologies. The ideal candidate will have strong expertise in data architecture, Lakehouse design, data engineering, and enterprise data platform modernization.

Key Responsibilities

• Design and implement end-to-end data architectures using Databricks and the Lakehouse platform.

• Architect scalable data pipelines using Apache Spark, PySpark, and Spark SQL.

• Implement and manage Delta Lake, Unity Catalog, and data governance frameworks.

• Design and optimize ETL/ELT pipelines for batch and streaming data processing.

• Integrate Databricks with cloud platforms such as AWS, Azure, or Google Cloud Platform.

• Establish data security, access controls, data quality, and compliance standards.

• Optimize workloads for performance, scalability, and cost efficiency.

• Implement CI/CD, automated testing, and deployment strategies for data engineering solutions.

• Collaborate with data engineers, business stakeholders, and enterprise architecture teams.

• Provide technical leadership, architectural guidance, and best practices for Databricks implementations.

Required Skills and Qualifications

• 10+ years of experience in data engineering, data architecture, or related fields.

• Strong hands-on experience with Databricks architecture and Lakehouse solutions.

• Expertise in Apache Spark, PySpark, Spark SQL, and Delta Lake.

• Experience with Unity Catalog, access management, and data governance.

• Strong knowledge of cloud data platforms, preferably Azure Databricks or AWS Databricks.

• Experience designing complex ETL/ELT pipelines and enterprise data integration solutions.

• Proficiency in Python and SQL.

• Experience with data modeling, data warehousing, and distributed processing.

• Knowledge of CI/CD pipelines, Git, and DevOps practices.

• Strong problem-solving, communication, and stakeholder management skills.

Preferred Qualifications

• Databricks certifications such as Databricks Certified Data Engineer Professional or Databricks Certified Data Engineer Associate.

• Experience with streaming technologies such as Kafka and Structured Streaming.

• Exposure to MLflow, feature engineering, and machine learning platform integration.

• Experience with Terraform or other infrastructure-as-code tools.

• Experience in enterprise-scale data platform migration and modernization