Databricks Data Engineer

Databricks Data Engineer

Location: New York , Onsite

Need Locals or nearby who can relocate

We are seeking an experienced Databricks Data Engineer to design, develop, and optimize enterprise-scale data pipelines on the Databricks platform. The ideal candidate will have strong expertise in Databricks, Delta Lake, Python, SQL, ETL/ELT, cloud platforms, and Sales Incentive Compensation (SIC) data processing.

• Design, develop, and maintain scalable data pipelines using Databricks, Delta Lake, and Delta Live Tables.
• Build and optimize data models, notebooks, workflows, and enterprise data pipelines.
• Develop data solutions supporting Sales Incentive Compensation (SIC) including quota management, commission calculations, attainment tracking, and payout reporting.
• Collaborate with Sales Operations, Finance, BI, and business stakeholders to translate business requirements into scalable data solutions.
• Manage and enhance Databricks architecture, Unity Catalog, and workflow automation.
• Optimize pipeline performance, troubleshoot issues, and ensure high data quality.
• Integrate Databricks with cloud services (Azure, AWS, or Google Cloud Platform) and data sources such as Azure Data Factory, Kafka, Event Hubs, or similar.
• Document data models, pipeline architecture, and technical specifications.

• Strong expertise in:
  • Databricks
  • Delta Lake
  • Delta Live Tables (DLT)
  • Unity Catalog
  • Databricks Workflows
  • Python and/or Scala
  • Advanced SQL
  • ETL/ELT development and data modeling
• Experience with Sales Incentive Compensation (SIC) processes including commissions, quotas, payouts, and attainment calculations.
• Experience integrating with Anaplan, Oracle ICM, Varicent (preferred).
• Experience with Azure, AWS, or Google Cloud Platform cloud platforms.
• Excellent analytical, communication, and stakeholder management skills.
• Databricks Data Engineer Certification is a plus.