Azure Databricks Engineer (PySpark) Tech Lead

# Job Description: Azure Databricks Engineer (PySpark)

• Tech Lead
• *Location:** Dallas, TX / Atlanta, GA
• *Work Mode:** 5 Days Onsite
• *Joining Requirement:** Day 1 Onsite

## Role Summary
We are seeking an experienced **Azure Databricks Engineer (PySpark)

• Tech Lead** to lead the end-to-end migration of enterprise data integration workloads from Informatica PowerCenter to Azure Databricks. The role requires strong hands-on expertise in PySpark, Azure Databricks, SQL, and data engineering, along with technical leadership, architecture design, and delivery ownership.

The ideal candidate will have proven experience analyzing complex Informatica mappings, migrating batch ETL pipelines, and modernizing Oracle-based data processing workloads into scalable, reliable, and high-performance Databricks solutions. Experience with Unity Catalog, Delta Live Tables (DLT), ADLS Gen2, and enterprise data platform modernization is essential.
##

Key Responsibilities

• Lead the end-to-end migration of data pipelines, workflows, and integration patterns from Informatica PowerCenter to Azure Databricks.
• Analyze existing Informatica PowerCenter 10.4.1 workflows, mappings, sessions, and batch ETL processes to define migration strategies and target-state architecture.
• Convert complex Informatica transformations, including lookups, joins, filters, routers, expressions, and aggregations, into equivalent PySpark and Spark SQL implementations.
• Analyze Oracle stored procedures, functions, packages, and database dependencies to determine whether to migrate, refactor, or recreate business logic in Databricks.
• Design and develop scalable ETL/ELT pipelines using PySpark, Python, and Spark SQL.
• Implement data engineering solutions using Azure Databricks Runtime 14.4 LTS, Unity Catalog, Delta Live Tables (DLT), and Azure Data Lake Storage Gen2 (ADLS Gen2).
• Modernize legacy ETL processes into optimized, maintainable, and reusable data engineering frameworks.
• Establish engineering standards covering architecture, coding practices, testing, deployment, performance optimization, and production readiness.
• Define data validation, reconciliation, and testing strategies to ensure functional parity between legacy Informatica workflows and Databricks pipelines.
• Optimize Spark jobs, data processing workloads, and SQL queries for performance, scalability, and cost efficiency.
• Collaborate with business stakeholders, solution architects, infrastructure, security, and operations teams to ensure successful migration delivery.
• Identify technical risks, dependencies, and migration challenges, and implement mitigation plans.
• Lead deployment, cutover, production stabilization, monitoring, and post-migration optimization activities.
• Mentor engineering teams on PySpark development, Databricks best practices, reusable frameworks, and modern data engineering methodologies.

## Required Technical Skills

• Strong hands-on experience with **Azure Databricks and PySpark**.
• Extensive experience with **Informatica PowerCenter**, including mapping analysis, workflow execution, transformations, sessions, and batch ETL processes.
• Proven experience migrating Informatica workloads to Databricks or other modern data engineering platforms.
• Strong programming skills in Python, PySpark, and Spark SQL.
• Strong SQL expertise and experience working with Oracle databases.
• Experience analyzing and refactoring Oracle stored procedures, functions, packages, and complex SQL logic.
• Hands-on experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• Experience with Unity Catalog, Delta Lake, and Delta Live Tables (DLT).
• Knowledge of ETL/ELT architecture, data modeling, data quality, and data reconciliation.
• Experience with Spark performance tuning, partitioning, caching, and distributed data processing.
• Experience designing data pipelines with appropriate error handling, logging, monitoring, and recovery mechanisms.
• Strong technical leadership, problem-solving, communication, and stakeholder management skills.

##

Preferred Qualifications

• Experience with enterprise-scale data modernization and legacy platform migration programs.
• Experience working with supply chain data, enterprise analytics, or complex business data integration environments.
• Familiarity with Azure DevOps, Git, CI/CD pipelines, and automated deployment practices.
• Experience with production support, incident resolution, release management, and operational readiness.
• Experience establishing reusable migration frameworks and engineering standards across multiple teams.

## Work Location and Availability

• **Locations:** Dallas, Texas, or Atlanta, Georgia.
• **Work Arrangement:** Fully onsite, five days per week.
• **Joining:** Must be available to work onsite from Day 1.

## Ideal Candidate Profile
The ideal candidate is a hands-on technical lead who can independently assess existing Informatica and Oracle workloads, develop migration strategies, implement equivalent PySpark pipelines, and lead the migration through testing, deployment, and production stabilization. Candidates should demonstrate both deep technical expertise and the ability to guide engineering teams through enterprise-scale data platform modernization.