Senior Data Engineer – Azure Databricks

Job Title: Senior Data Engineer
Location: Remote
Employment Type: Contract

Job Summary

We are looking for an experienced Senior Data Engineer with strong expertise in Python, PySpark, Apache Spark, Databricks, Azure Data Factory (ADF), and SQL. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines and ETL/ELT processes in a cloud-based data engineering environment.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python, PySpark, Apache Spark, and Databricks.

• Develop and optimize data processing workflows using Azure Databricks and Azure Data Factory (ADF).

• Write complex and optimized SQL queries for data extraction, transformation, validation, and analysis.

• Build batch and large-scale data processing solutions using Apache Spark/PySpark.

• Develop reusable data engineering frameworks and components using Python.

• Create and maintain ADF pipelines, datasets, triggers, and integrations.

• Perform data transformation, cleansing, validation, and reconciliation across multiple data sources.

• Work with Unix/Linux shell scripting for automation, scheduling, and production support.

• Use Control-M for enterprise job scheduling, monitoring, and batch processing.

• Troubleshoot production data pipelines and resolve performance, data quality, and integration issues.

• Optimize Spark jobs, Databricks workloads, SQL queries, and ETL processes for performance and scalability.

• Collaborate with architects, analysts, developers, and business teams to understand data requirements.

• Follow data engineering best practices for performance, reliability, scalability, and data quality.

• Support deployment, monitoring, and production operations of data pipelines.

Required Skills

• Strong hands-on experience with Python

• Strong experience with PySpark and Apache Spark

• Hands-on experience with Databricks

• Strong experience with Azure Data Factory (ADF)

• Advanced SQL skills

• Experience with Unix/Linux Shell Scripting

• Experience with Control-M

• Strong understanding of ETL/ELT concepts

• Strong data engineering and data pipeline development experience

• Experience working with large-scale datasets and distributed data processing

Preferred / Optional Skills

• AWS Cloud experience is preferred.

• Experience with AWS services such as S3, Glue, EMR, Redshift, or Lambda is a plus.

• Experience working in multi-cloud environments is an advantage.

• Experience with CI/CD and DevOps practices for data engineering is a plus.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

• 6+ years of experience in Data Engineering with strong hands-on development experience.

• Strong problem-solving and communication skills.

• Ability to work independently as well as collaboratively in a technical team environment.