Senior Data Engineer (PySpark)

Position: Senior Data Engineer (PySpark)

Location: New York City, NY

Duration: Full Time

Qualifications & Requirements

• Minimum 5 to 7 years of data engineering experience, with at least 3+ years of hands-on Databricks experience in production environments.
• Migration Expertise

Proven experience in migrating on-premises data systems to cloud platforms.

• Technical Skills

Strong proficiency in Python, PySpark, and SQL. Experience in converting legacy SQL dialects to Spark SQL is a plus.

• Cloud Knowledge

Familiarity with cloud computing concepts, preferably within Azure ecosystems. Experience integrating cloud functions or runtime services with data platforms is desirable.

• Best Practices & Tools

Solid understanding of CI/CD, data quality frameworks, and orchestration tools such as Databricks Workflows, Airflow, or dbt.

Preferred Skills

• Experience with large-scale data modernization programs
• Exposure to financial services or regulated data environments
• Strong problem-solving and stakeholder communication skills
• Investment banking background would be added advantage.