Lead Azure Data Engineer (In-Person Interview in San Jose, CA)

Sr Azure Databricks Engineer

Location- - Milpitas, CA- 95035 (In-Person Interview and Must be Local to Bay Area, CA)

Rate- $72/hr

Job Summary

We are looking for a skilled Databricks Engineer with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
• Build ETL/ELT workflows for batch and streaming data processing.
• Develop solutions using PySpark, Spark SQL, and Delta Lake.
• Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.
• Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.
• Implement CI/CD pipelines and deployment automation for Databricks workloads.
• Ensure data quality, security, governance, and compliance.
• Monitor, troubleshoot, and optimize production data pipelines.
• Document technical solutions and follow engineering best practices.

Required Skills

Core Technologies

• Databricks Lakehouse Platform
• Apache Spark
• PySpark
• Spark SQL
• Delta Lake
• Python
• SQL

Cloud Platforms (one or more)

• Microsoft Azure (preferred)
• AWS
• Google Cloud Platform

Azure Technologies (Preferred)

• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS Gen2)
• Azure Synapse Analytics
• Azure Key Vault
• Azure DevOps

Data Engineering

• Data Warehousing
• Data Modeling
• ETL/ELT Development
• Batch Processing
• Streaming (Kafka/Event Hubs)
• Data Lake Architecture

DevOps & Version Control

• Git
• Azure DevOps / GitHub
• CI/CD Pipelines

Preferred Qualifications

• Experience with Unity Catalog.
• Knowledge of Databricks Workflows and Jobs.
• Hands-on experience with Delta Live Tables (DLT).
• Exposure to MLflow is an added advantage.
• Experience with data governance and security best practices.
• Familiarity with Infrastructure as Code (Terraform) is a plus.

Educational Qualification

• Bachelor''s or Master''s degree in Computer Science, Information Technology, Engineering, or a related field.

Preferred Certifications

• Databricks Certified Data Engineer Associate
• Databricks Certified Data Engineer Professional
• Microsoft Certified: Azure Data Engineer Associate (DP-203)
• Azure Fundamentals (AZ-900)

Good to Have

• Experience with real-time analytics.
• Knowledge of Lakehouse architecture.
• Experience with Agile/Scrum methodologies.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.

Mandatory Skills

• Databricks
• PySpark
• Spark SQL
• Delta Lake
• Python
• SQL
• Azure/AWS/Google Cloud Platform (at least one cloud platform)
• ETL/ELT Development
• Data Lake Architecture

Nice to Have

• Unity Catalog
• Delta Live Tables (DLT)
• MLflow
• Kafka/Event Hubs
• Azure Data Factory
• Terraform
• Azure DevOps/GitHub Actions