Haystack
← Back to Jobs
Technology

Lead Data Engineer

SG Analytics Inc.New York, NY🇺🇸United StatesPosted 10 Jul 2026

Why This Role Stands Out

Leverage your expertise in Databricks, PySpark, and AWS to architect and lead enterprise-scale data platforms within the dynamic New York media industry, mentoring a talented team and shaping impactful media measurement solutions. If you're a driven technical leader eager to drive innovation and deliver complex projects in a collaborative environment, this role offers exceptional growth opportunities.

Quick Overview

Work Type
On Site
Level
Mid Senior

Job Description

About the Role
We are looking for an experienced Lead Data Engineer with deep expertise in Databricks, PySpark, and AWS. The ideal candidate will have a strong understanding of media measurement datasets (Nielsen) and a proven ability to lead multiple concurrent projects across onsite and offshore teams. This role requires a senior technical leader to define architecture, manage delivery, and mentor teams within the fast-paced New York media industry.
 
Key Responsibilities
  • Technical Leadership: Architect and deliver enterprise-scale data platforms using Databricks, PySpark, Delta Lake, and AWS. Lead the development of media analytics pipelines, including the ingestion and modeling of Nielsen data.
  • Project & Delivery Management: Manage scope, timelines, and risks for multiple simultaneous projects. Collaborate with New York-based stakeholders and drive sprint delivery.
  • Onsite–Offshore Coordination: Act as the primary technical lead onsite, coordinating with offshore teams to ensure alignment, quality, and continuity.
  • Team Mentorship: Mentor junior and mid-level engineers through code reviews and architectural guidance.
Required Skills
  • Databricks: Delta Lake, Workflows, Unity Catalog, Notebooks, and SQL.
  • Data Processing: Strong PySpark and distributed computing experience.
  • AWS: S3, EMR, Glue, Lambda, Step Functions, Kinesis, and IAM.
  • Domain Expertise: Proven experience handling Nielsen datasets (National/Local TV ratings, Digital Ad ratings, Panels).
  • Leadership: Experience leading large data engineering programs and managing senior stakeholders.
Good to Have
  • Knowledge of Snowflake, Airflow, or Dagster.
  • Infrastructure as Code (Terraform / CloudFormation).
  • Exposure to ad-tech ecosystems (Comscore, Kantar, IAS, Moat) or ML feature pipelines.
 
Soft Skills
  • Strong decision-making and ownership mindset.
  • Exceptional communication skills for interacting with senior business stakeholders.
  • Strategic, problem-solving, and delivery-focused approach.

Skills

SQL
AWS
Snowflake
Airflow
CloudFormation
Databricks
Terraform
Unity

Similar jobs

Apply now