Lead Data Engineer – Macro Financial Data & Analytics

Lead Data Engineer – Macro Financial Data & Analytics 

Location – Parsippany NJ - (Locals to NJ only)

Duration – Long term

About the Role

We are seeking an experienced Lead Data Engineer with a blend of approximately 70% data engineering and 30% analytical responsibilities. This individual will have strong proficiency in modern cloud-native data platforms, large-scale data processing, and the ability to contribute meaningfully to analytical and data science workflows.

The ideal candidate combines deep Databricks and PySpark expertise with finance or payroll domain experience and the analytical fluency to partner closely with data scientists on macroeconomic and financial data research.

This role will work across production pipeline development, platform optimization, exploratory data analysis, and close collaboration with data scientists, economists, and business stakeholders to build and maintain scalable data platforms for financial analytics and research.

The right candidate will be comfortable moving fluidly between engineering and analytical work — building production ETL pipelines, exploring and profiling new datasets, and helping translate complex analytical requirements into scalable solutions.

Required Qualifications

·         Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, Statistics, Economics, Finance, or a related field.

·         5 years of experience in Data Engineering or Data Platform development.

·         Finance or payroll domain experience, including familiarity with payroll data structures, pay-period logic, compensation and deduction relationships, or financial-services data environments.

·         Experience handling large-scale datasets, including billions of records, multi-terabyte environments, and time-series data.

·         Strong Databricks proficiency with a deep understanding of:

o    Unity Catalog, including governance, access patterns, and catalog/schema design.

o    Delta Lake internals, including optimization, clustering, change data feed, and versioning.

o    Databricks Workflows, including orchestration, dependencies, and monitoring.

o    Databricks Asset Bundles or equivalent deployment frameworks.

·         Experience participating in architecture-level design decisions and evaluating technical tradeoffs.

·         Strong proficiency in:

o    Python

o    SQL

o    PySpark

o    Data Modeling

o    ETL/ELT Development

·         Analytical fluency, including exploratory data analysis, basic statistical concepts, distributions, correlations, time-series patterns, and feature engineering.

·         Proficiency with pandas and NumPy for ad hoc analytical work alongside production PySpark.

·         CI/CD implementation experience using tools such as Bitbucket Pipelines, Jenkins, and automated deployment frameworks.

·         Experience with AI-assisted development tools such as GitHub Copilot, Amazon Q, Kiro, or equivalent.

·         Experience implementing data quality and validation frameworks.

Technical Skills

Programming: Python, SQL, PySpark, Scala preferred

Data Engineering & Analytics: Databricks, Apache Spark, Delta Lake, Unity Catalog, Databricks Asset Bundles, Kafka

Databases: SQL Server, PostgreSQL, Delta Tables, NoSQL databases

Visualization & Reporting: Power BI, Databricks Dashboards, Python visualization libraries including Matplotlib and Plotly

DevOps & Automation: Git, Bitbucket Pipelines, Jenkins, Terraform, CI/CD pipelines, Databricks Asset Bundles