Data Engineer Role

Role- Data Engineer

Location- New York

Job Description

We are seeking an experienced Data Engineer to join our team and build robust, scalable data pipelines. In this role, you will:

Design and implement scalable PySpark data pipelines for batch and streaming workloads

Optimize Spark jobs and queries for performance and cost efficiency

Build and maintain ETL/ELT processes following data engineering best practices

Troubleshoot and resolve complex data pipeline and processing issues

Collaborate with data teams to ensure data quality and reliability

Top Skills

Databricks Platform Experience

Data Engineering & Pipeline Development

Advanced ETL/ELT pipeline design and development

Incremental data processing patterns (CDC, SCD Type 2)

Data Processing & Optimization

Spark optimization techniques (partitioning, bucketing, caching, broadcast joins)

Required Technical Skills

Databricks & Spark Proficiency: 3+ years of hands-on experience building data pipelines in Databricks; deep understanding of Spark fundamentals, transformations, actions, and performance optimization techniques including partitioning, caching, and resource management

Advanced PySpark and SQL Skills: Expert-level proficiency writing production-quality PySpark code and complex SQL queries for data transformation, aggregation, and analysis; experience with DataFrame API, Spark SQL, and UDFs; strong understanding of lazy evaluation and execution plans

Data Engineering & ETL/ELT: Proven experience building and maintaining production data pipelines; hands-on experience with incremental data loading, change data capture (CDC), and slowly changing dimensions; experience handling data quality issues and implementing data validation frameworks

Cloud & Big Data Technologies: Strong proficiency with AWS services (S3, EC2, IAM, Glue, Athena); experience working with large-scale distributed data processing; familiarity with data formats (Parquet, Delta, JSON, Avro) and compression techniques

DevOps & CI/CD: Experience with version control (Git) and CI/CD pipelines using GitLab, GitHub Actions, or similar tools; familiarity with testing data pipelines and deployment automation; experience with Databricks Repos and workspace-level integrations

Data Governance: Understanding of data lineage, cataloging, and metadata management; experience implementing data quality checks and monitoring; knowledge of data privacy and security best practices in cloud environments (nice to have)

If interested please email updated resume to