Data Engineer_Azure Databricks

Data Engineer – Azure Databricks

Overview

Design, build, and maintain scalable batch and streaming data pipelines, ETL workflows, and AI solutions within the Microsoft Azure and Fabric ecosystems. This role bridges cloud data engineering, advanced analytics, and machine learning infrastructure.

Core Responsibilities:

Data Engineering & Pipelines

• Build scalable ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, and Fabric Data Factory.
• Handle full, incremental, and Change Data Capture (CDC) data ingestion.
• Manage unstructured and structured data at petabyte scale across ADLS Gen2 and OneLake.

 Architecture & Optimization

• Implement Lakehouse and Medallion Architectures using Delta Lake (ACID, schema evolution, time travel).
• Conduct deep performance tuning of Apache Spark internals (partitioning, caching, cluster optimization).
• Optimize complex SQL/T-SQL queries, window functions, and indexing strategies.

Governance & DevOps

• Enforce security using Unity Catalog, RBAC, row/column-level security, and data masking.
• Manage CI/CD pipelines via Git, Azure DevOps/GitHub Actions, and Databricks Asset Bundles (DAB).
• Deploy Infrastructure as Code (IaC) using Terraform, Bicep, or ARM templates.

 AI & Analytics Integration

• Build infrastructure for AI applications using Azure Machine Learning, Cognitive Services, and RAG.
• Collaborate with data scientists to support MLOps, Copilot Studio, and automated testing frameworks.
• Power downstream analytics by developing Power BI dashboards and advanced DAX models.

Technical Skills

Required Qualifications

• Education: Bachelor’s degree in computer science, IT, or a related quantitative field.
• Experience: 3+ years of professional cloud data engineering experience.
• Methodologies: Strong proficiency in Agile/Scrum environments using Jira.
• Soft Skills: Excellent communication skills to bridge technical delivery with business stakeholder requirements.