Lead Data Engineer Databricks

Job Title: Lead Data Engineer Databricks Location: New York City, NY | Hybrid (3 days per week onsite)
Ideal Experience Range: 6-12 Years
Interview Process: 3 Rounds with One being onsite in NY/NJ
Client: Media Streaming
Key requirements for this role:

• Hands-on Databricks experience within the last 2 years, with working knowledge of its features
• Prior data engineering experience, including building pipelines and integrating multiple data sources
• Knows data modeling

Role Overview
Client is seeking a Lead Data Engineer with deep hands-on expertise in Databricks, SQL, Python, and PySpark to design, develop, and optimize scalable data engineering solutions.
This role offers an opportunity to work on enterprise-scale data platforms supporting analytics, digital products, and emerging AI use cases. The candidate will drive Databricks engineering best practices, solve challenging technical problems, and provide technical leadership across multiple initiatives.
Key Responsibilities

• Design and develop scalable data pipelines on Databricks, following Medallion Architecture.
• Build and optimize data ingestion, transformation, and orchestration workflows for batch and incremental processing.
• Optimize data processing workloads, SQL queries, and compute configurations for performance and cost efficiency.
• Ensure data quality, observability, lineage, and reliability across production pipelines, including troubleshooting complex issues.
• Provide hands-on technical leadership through solution design, code reviews, reusable engineering patterns, and mentoring other engineers.

Required Qualifications

• 6+ years of Data Engineering experience, with substantial hands-on experience delivering production solutions on Databricks.
• Strong proficiency in SQL, Python, and PySpark, including distributed data processing and performance optimization.
• Strong understanding of Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and workflow orchestration.
• Familiarity with Git, CI/CD, and production deployment practices.
• Strong troubleshooting, problem-solving, and communication skills.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional certification.
• Experience working with AWS cloud services, particularly S3 and cloud-based data integrations.
• Experience with advanced ingestion patterns such as Auto Loader, Structured Streaming, or Change Data Capture (CDC).
• Familiarity with Databricks Asset Bundles or Apache Airflow.
• Experience with large-scale media datasets: streaming, clickstream, or advertising data.

Thanks And Regards,

Amit Lakhotia