Databricks Architect

Key Responsibilities

• Design and implement end-to-end data architecture using Databricks Lakehouse platform
• Define data engineering best practices and build scalable ETL/ELT pipelines
• Architect solutions leveraging Apache Spark (PySpark/Scala) and Databricks workflows
• Implement Delta Lake for data reliability, performance, and governance
• Collaborate with stakeholders to understand business requirements and translate them into technical solutions
• Optimize performance, cost, and scalability of data pipelines
• Provide technical leadership, mentoring data engineers and developers
• Ensure data security, governance, and compliance standards are met
• Integrate Databricks with cloud services (Azure, AWS, or Google Cloud Platform) and other enterprise systems
• Support real-time and batch data processing use cases

Required Skills & Qualifications

Technical Skills

• Strong hands-on experience with Databricks Platform
• Expertise in Apache Spark (PySpark / Scala / SQL)
• Experience with Delta Lake, Unity Catalog, and Lakehouse architecture
• Proficiency in data pipelines, ETL/ELT frameworks
• Strong experience in SQL and data modeling
• Hands-on experience with cloud platforms:
  • Azure (ADF, ADLS, Synapse) OR
  • AWS (S3, Glue, EMR) OR

Data Engineering Concepts

• Data warehousing and data lake solutions
• Batch & real-time data processing
• Data governance and security frameworks
• Performance tuning and optimization

Tools & Technologies

• Databricks Workflows / Jobs
• Airflow / ADF / Prefect (any orchestration tool)
• CI/CD pipelines (Azure DevOps, GitHub Actions)
• Streaming tools like Kafka (preferred)

Preferred Qualifications

• Databricks Certifications (e.g., Databricks Certified Professional / Associate)
• Experience in data migration to Databricks