Azure Data lead - Python, Pyspark, Databricks, ADF, Data Lake

Location: New York, NY

Experience: 7-12 Years

Primary Skills: Azure Data Factory (ADF), Azure Databricks, SQL, Oracle PL/SQL, Python

Certifications: Relevant certifications in Azure Data Factory, Azure Databricks, SQL, Oracle, or Python would be a plus.

Other Requirement:

Azure Data Lead - Python, PySpark, Databricks, ADF, Data Lake

Azure Senior Data Lead leads modernization of Python applications into scalable PySpark solutions on Azure Databricks.

Lead the modernization and migration of existing Python object-oriented applications into scalable PySpark and Spark SQL data-processing solutions on Azure Databricks - bringing a strong blend of software engineering, data engineering, cloud architecture, and performance optimization.

• Analyze existing Python OOP applications and redesign single-node processing logic for distributed Spark execution.
• Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules.
• Implement Delta Lake solutions using the Bronze Silver Gold architecture.
• Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
• Implement data quality, reconciliation, validation, and monitoring frameworks.
• Optimize Spark jobs using partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, and Delta optimization.
• Benchmark converted applications against original Python implementations.

• Python (expert), OOP, and advanced Python design patterns
• PySpark, Spark SQL, and SQL
• Azure Databricks, Azure Data Factory, ADLS Gen2
• Apache Spark, Delta Lake, Data Lakehouse architecture, distributed computing

• Python
• Azure Databricks
• Azure Data Factory (ADF)
• MS SQL
• Oracle PL/SQL

• PySpark
• Certifications in Azure Data Factory, Azure Databricks, SQL, Oracle, or Python

Senior data engineering leader with proven delivery of large-scale Databricks modernization programs.

Expected outcome: Existing Python applications converted into scalable, cost-efficient, enterprise-grade data solutions on Azure Databricks with proven performance parity.