Azure Data Engineer

Job Title: Azure Data Engineer

Location: Greenfield, IN Onsite

Duration: Direct Hire

Key Technical Skills & Expertise

·        Databricks & PySpark Data Engineering: End-to-end Lakehouse architecture design (Bronze/Silver/Gold), batch and near-real-time ingestion (ERP, APIs, Relational, NoSQL, On-prem), Spark tuning (RDD/DataFrame APIs, partitioning, Parquet/Delta), and Databricks workspace management.

·        Data Governance & Security (Unity Catalog & ADLS Gen2): Granular access control (RBAC/ACLs), dynamic data masking, lineage tracking, Azure AD/Entra ID integration, and lifecycle management on ADLS Gen2 hierarchical namespaces.

·        Orchestration & Data Pipelines: Designing modular, parameterized pipelines using Azure Data Factory (ADF) and Databricks Workflows with custom activities, integration runtimes, and robust error-handling/retry strategies.

·        Azure Cloud & Infrastructure: Architecting secure Azure environments utilizing VNets, Private Endpoints, Key Vaults, Managed Identities, and Azure Monitor for performance tuning and cost optimization.

·        Python Development & Software Engineering: Modular, production-grade Python/PySpark development, dependency management (pip/Poetry/Conda), unit testing (pytest/unittest), and code quality enforcement (SonarQube/pylint).

·        Database & Metadata Management: Complex T-SQL querying, stored procedures, and metadata layer maintenance on Azure SQL Database.

·        DevOps & CI/CD: Automated deployment pipelines using Azure DevOps and Infrastructure as Code (IaC) via Terraform, Bicep, or ARM templates with Git workflows.

·        Technical Leadership & Delivery: Authoring HLD/LLD design documents, establishing architecture standards, leading code reviews, mentoring engineers, and effectively bridging communication between business stakeholders and delivery teams

Educational Qualification*          

Any Bachelor's Degree

Experience Range :12+ Years

Tagline/Tech Stack Snapshot –

Azure Databricks | PySpark | ADF | ADLS Gen2 | Azure SQL | Python | Unity Catalog | Delta Lake | Azure Cloud | CI/CD | Git | Azure DevOps

Role Summary - (To be filled by Practice /DO)

We are looking for an experienced Data Engineering Tech Lead with strong expertise in Azure Databricks, PySpark, Azure Data Factory (ADF), ADLS Gen2, Azure SQL, Python, and Azure Cloud. The role will lead the design, development, optimization, and maintenance of scalable enterprise data engineering and Lakehouse solutions, including batch and near-real-time data processing.

Primary (Must have skills)*

4+ years of experience in Azure Databricks with PySpark.
2+ years of experience in Databricks workflow & Unity catalog.
3+ years of experience in ADF (Azure Data Factory).
3+ years of experience in ADLS Gen 2.
3+ years of experience in Azure SQL.
5+ years of experience in Azure Cloud platform.
2+ years of experience in Python programming & package builds

Secondary Skills  (Good to have)

Good to have Azure Entra/AD skills and GitHub Actions.
Good to have orchestration experience using Airflow, Dagster, LogicApp.
Good to have expereince working on event-driven architectures using Kafka, Azure Event Hub.
Good to have exposure on Google Cloud Pub/Sub.
Good to have experience developing and maintaining Change Data Capture (CDC) solutions preferrably using Debezium.
Good to have hands-on experience on data migration projects specifically involving Azure Synapse and Databricks Lakehouse.
Good to have eperienced in managing cloud storage solutions on Azure Data Lake Storage . Experience with Google Cloud Storage will be an advantage.