Lead Databricks Data Engineer

Location: New York, NY – Hybrid (3 Days Onsite/Week)
Experience: 6+ Years
Employment Type: Full Time

Job Description

We are looking for an experienced Lead Databricks Data Engineer to design, develop, and optimize scalable data engineering solutions. The ideal candidate must have strong hands-on experience with Databricks, PySpark, Delta Lake, Snowflake, and modern cloud data platforms.

Minimum 2+ years of genuine, hands-on Databricks experience within the last 2 years is mandatory. Candidates must be able to demonstrate recent project experience and strong technical expertise.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.

• Build and optimize data pipelines using Delta Lake and Medallion Architecture (Bronze, Silver, Gold).

• Develop and maintain data engineering frameworks and reusable components.

• Integrate Databricks with Snowflake and cloud-based data platforms.

• Implement data orchestration using Apache Airflow and Databricks Workflows.

• Optimize Spark jobs, SQL queries, cluster performance, and data processing costs.

• Implement data quality, governance, monitoring, and observability practices.

• Collaborate with architects, business stakeholders, and engineering teams to define technical solutions.

• Provide technical leadership, code reviews, architecture guidance, and mentorship to data engineering teams.

Required Skills

• 6+ years of experience in Data Engineering.

• 2+ years of recent, hands-on Databricks experience within the last 2 years (Mandatory).

• Strong experience with Azure Databricks or Databricks on AWS.

• Strong programming skills in Python, PySpark, and SQL.

• Hands-on experience with Delta Lake and Medallion Architecture.

• Strong knowledge of Apache Spark and performance optimization.

• Experience with Snowflake and data warehousing concepts.

• Experience with Apache Airflow or similar orchestration tools.

• Knowledge of cloud platforms (Azure, AWS, or Google Cloud Platform).

• Experience in technical leadership, architecture, and mentoring.

Preferred Qualifications

• Experience with Unity Catalog, Delta Live Tables, and Databricks Workflows.

• Knowledge of streaming technologies such as Kafka.

• Experience with CI/CD, DataOps, and Infrastructure as Code.

• Exposure to data governance, security, and data quality frameworks.

• Experience in enterprise-scale data modernization projects.

Important Requirement

Candidates must have genuine recent hands-on Databricks development experience. Profiles with only theoretical knowledge, limited migration exposure, or primarily Microsoft Fabric/Snowflake experience without recent Databricks implementation will not be considered.

Interested candidates can share their updated resume along with current location, work authorization, and availability.