Lead Data Engineer Pipeline Engineering Focus

Lead Data Engineer - Pipeline Engineering Focus

Client: Mastercard

121 Bloor st E, Toronto, Canada (onsite 5days)

Position Overview

We are looking for a Lead Data Engineer to build, operate, and scale the data pipelines and processing frameworks that power downstream analytics and data products. This role is execution-focused: the primary mandate is engineering excellence - reliable ETL/ELT, strong data quality practices, and operational rigor - rather than direct business stakeholder management. As the subject matter expert on data modeling within the team, you'll also lead engineering activities and mentor other engineers through hands-on technical guidance.

Key Responsibilities

Pipeline Engineering & Operations

• Build and maintain scalable, reliable data pipelines and processing frameworks across the big data ecosystem (Spark, Databricks, Hadoop, PySpark).
• Own the full ETL/ELT lifecycle - ingestion, transformation, aggregation, and processing of large-scale datasets - with a focus on pipeline reliability and automation.
• Design and maintain workflow orchestration using Airflow, ensuring pipelines run reliably and recover gracefully from failure.
• Work with modern table formats (Delta, Iceberg) to support scalable, versioned, and performant data storage.

Data Quality & Governance

• Establish and enforce data quality standards: validation, monitoring, and governance practices across pipelines.
• Ensure secure, compliant data usage in line with data governance and privacy requirements.
• Serve as the team's subject matter expert on data modeling, setting standards for schema design and data structure.

Engineering Leadership

• Lead engineering activities and set technical direction through hands-on expertise, not just process oversight.
• Mentor engineers on pipeline design, performance optimization, and engineering best practices.
• Champion CI/CD, automation, and strong source control practices across the data engineering workflow.

Cross-Functional Collaboration

• Partner with Data Science, Product, Analytics, and Infrastructure/Engineering teams to understand pipeline and data requirements.
• Support analytics and downstream consumers by ensuring pipelines deliver clean, reliable, well-structured data - with limited direct business-facing engagement.

Innovation

• Stay current with and adopt emerging data engineering technologies and practices to continuously improve pipeline architecture and performance.

Required Skills & Experience

• Strong, hands-on experience with Spark, Databricks, Hadoop, PySpark.
• Experience with Airflow or similar orchestration tools for pipeline scheduling and dependency management.
• Experience with modern lakehouse table formats such as Delta Lake and/or Apache Iceberg.
• Proven track record designing and operating pipeline architecture at scale - not just writing individual jobs.
• Strong background in data quality engineering: validation frameworks, monitoring, alerting, and data governance standards.
• Solid grounding in cloud-native data engineering practices.
• Experience with CI/CD, automation, and source control as part of the data engineering SDLC.
• Demonstrated ability to lead through technical expertise and mentor other engineers.
• Strong communication skills - able to clearly explain technical designs and trade-offs to engineering peers.

Not required for this role: GenAI/LLM experience, machine learning, or heavy business-facing analytics work - this role is scoped for pipeline engineering depth over broad business engagement.

Ideal Candidate Profile

A hands-on Lead Data Engineer who is energized by building and operating high-quality, production-grade data pipelines - someone who treats data quality and reliability as first-class engineering problems, and who leads by doing rather than by delegating.