Data Engineer II

Design and optimize scalable data pipelines and workflows for large datasets. Use Apache Spark, ADF, Airflow, and advanced SQL to support batch and streaming data systems while ensuring data quality, governance, and security. Collaborate with analysts, data scientists, and stakeholders to deliver reliable data products and insights.

Responsibilities

• Design and optimize scalable data pipelines and workflows for large datasets.
• Implement data processing and scheduling using Apache Spark, ADF, and Airflow.
• Write and optimize advanced SQL queries for reporting and applications.
• Maintain data quality, governance, and security through automated testing and monitoring.
• Develop infrastructure for real-time and batch processing workflows.
• Collaborate with stakeholders and provide technical guidance to junior data engineers.

Required Skills

• SQLPython
• Java
• Scala
• Apache Spark
• Azure Data Factory (ADF)
• Apache Airflow
• AWS Glue
• Data lake and lakehouse technologies
• Batch and streaming data processing