Lead Data Engineer (Data Engineering Tower Lead)

Location: Irvine, CA
Experience: 8–14 Years

Position Overview

We are seeking an experienced Lead Data Engineer to oversee enterprise-scale data engineering initiatives and manage multiple delivery teams. This role will be responsible for end-to-end data platform delivery, operational excellence, platform reliability, and stakeholder engagement while driving scalable data solutions that support critical business functions.

The ideal candidate will possess strong expertise in modern data engineering platforms, workflow orchestration, cloud-native architectures, and large-scale data processing environments.

Required Skills & Qualifications

Data Engineering & Big Data Technologies

• Strong hands-on experience with:
  • Databricks
  • Apache Spark
  • Distributed Data Processing
  • Performance Tuning & Optimization
• Deep understanding of:
  • Data Engineering Architecture
  • ETL/ELT Frameworks
  • Data Warehousing Concepts
  • Large-Scale Data Platforms
• Experience building scalable and reliable data pipelines.

Workflow Orchestration

• Expertise in:
  • Apache Airflow
  • Astronomer
  • Workflow Automation
  • DAG Design & Management
• Experience managing complex data workflows and dependencies.

DevOps & Platform Engineering

• Strong knowledge of:
  • CI/CD Pipelines
  • DevOps Practices
  • Infrastructure as Code (IaC)
  • Automation Frameworks
• Experience supporting cloud-native and enterprise data platforms.

Platform Operations & Reliability

• Experience with:
  • Platform Monitoring
  • Incident Management
  • Alerting & Observability
  • Production Support
  • Operational Readiness
• Proven ability to maintain highly available and resilient data environments.

Leadership & Stakeholder Management

• Experience leading multiple data engineering teams, pods, or delivery streams.
• Strong communication and stakeholder management skills.
• Ability to coordinate across onshore, offshore, and distributed global teams.
• Experience managing project risks, dependencies, and escalations.

Preferred Industry Experience

• Asset Management
• Investment Management
• Capital Markets
• Portfolio Analytics Platforms
• Market Data Platforms
• Financial Services Data Ecosystems

Key Responsibilities

Team Leadership

• Lead and manage multiple data engineering teams and delivery pods.
• Provide technical direction, mentorship, and delivery oversight.
• Foster collaboration across geographically distributed teams.

Data Platform Delivery

• Own end-to-end delivery of enterprise data engineering solutions.
• Drive development of scalable, high-performance data pipelines.
• Ensure platform stability, reliability, and operational excellence.

Platform Operations

• Establish and maintain monitoring, alerting, and incident management processes.
• Ensure platform availability and production support readiness.
• Drive continuous improvements in platform performance and reliability.

Stakeholder Engagement

• Partner with business, technology, and leadership teams to understand requirements and priorities.
• Manage cross-functional dependencies and project communications.
• Provide regular updates on delivery progress, risks, and operational metrics.

Governance & Best Practices

• Define and implement standards for:
  • Data Engineering Best Practices
  • Platform Governance
  • Operational Readiness
  • Performance Optimization
  • Automation & Deployment Processes
• Promote engineering excellence and continuous improvement across teams.

Delivery & Operational Excellence

• Ensure adherence to SLAs and operational objectives.
• Drive data quality, scalability, and performance initiatives.
• Lead issue resolution, root cause analysis, and production support activities.

Preferred Leadership Skills

• Strong leadership and people management capabilities.
• Excellent communication and stakeholder engagement skills.
• Experience leading enterprise-scale data engineering programs.
• Strong analytical, problem-solving, and decision-making abilities.
• Ability to operate effectively in fast-paced, highly collaborative environments.
• Experience managing globally distributed teams and complex delivery programs.

Keywords

Data Engineer Lead, Data Engineering Manager, Databricks, Apache Spark, Apache Airflow, Astronomer, ETL, ELT, Data Pipelines, Data Platform, CI/CD, DevOps, Infrastructure as Code, Cloud Data Engineering, Data Architecture, Big Data, Platform Reliability, Incident Management, Financial Services, Asset Management, Investment Management, Capital Markets, Team Leadership.