Data Pipeline Engineer (Databricks + AWS Glue)

Experience: 5–10 Years
Employment Type: Full-Time

This second requirement is much more specifically centered around Databricks + AWS Glue + PySpark + Python + SQL + MongoDB, so the knockout questions should be stricter around those technologies.

About the Role

We are looking for an experienced Data Pipeline Engineer with strong hands-on expertise in Databricks, AWS Glue, PySpark, Python, SQL, and MongoDB.

The engineer will design, develop, and optimize scalable cloud-based ETL and data pipelines for processing large volumes of structured and semi-structured data.

Responsibilities

• Design and develop scalable ETL and data pipelines using Databricks and AWS Glue.
• Build data ingestion and transformation workflows.
• Develop large-scale data processing solutions using PySpark, Python, and SQL.
• Integrate data from multiple sources including MongoDB and relational databases.
• Implement data-quality, validation, and reconciliation processes.
• Optimize Spark and Databricks workloads for performance and scalability.
• Build reliable cloud-native data pipelines on AWS.
• Support production deployments, monitoring, and troubleshooting.
• Collaborate with architects, application teams, analysts, and DevOps teams.
• Follow Git-based development and Agile delivery practices.