Haystack
← Back to Jobs
Technology

Data Engineer

Learn Beyond Consulting LLCUnited States🇺🇸United StatesPosted 27 Jul 2026

Quick Overview

Work Type
Hybrid
Level
Mid Senior

Job Description

Job Title: Databricks Data Engineer
Location: Remote
Employment Type: Contract
Job Summary
We are looking for an experienced Databricks Data Engineer with 7–8 years of overall IT experience to support a SQL Server to Databricks migration initiative. The ideal candidate will have strong expertise in Databricks, PySpark, Spark SQL, SQL Server migration, stored procedure conversion, notebook optimization, and query performance tuning. Exposure to Databricks AI features is a plus.
Required Skills
  • 7–8 years of overall IT experience with strong Data Engineering expertise
  • Hands-on experience with Databricks
  • Strong proficiency in PySpark and Spark SQL
  • Experience migrating workloads from SQL Server to Databricks
  • Experience converting SQL Server Stored Procedures into Databricks notebooks or PySpark code
  • Strong knowledge of SQL query optimization and Databricks notebook performance tuning
  • Experience with Delta Lake and modern data engineering practices
  • Strong ETL/ELT development experience
  • Excellent debugging and performance tuning skills
Preferred Skills
  • Experience with Azure Databricks, AWS, or Google Cloud Platform
  • Knowledge of Unity Catalog
  • Exposure to Databricks AI features such as MLflow, Mosaic AI, AI Functions, or Vector Search
  • Experience with Azure Data Factory, Airflow, or similar orchestration tools
Key Responsibilities
  • Migrate SQL Server workloads to Databricks.
  • Convert SQL Server stored procedures into scalable PySpark or Spark SQL solutions.
  • Optimize Databricks notebooks and Spark SQL queries for performance and scalability.
  • Design, develop, and maintain robust ETL/ELT data pipelines.
  • Work closely with architects and cross-functional teams to ensure a successful migration.
  • Troubleshoot and resolve data processing and performance issues.
  • Follow coding standards, best practices, and documentation guidelines.
Qualifications
  • Bachelor''s degree in Computer Science, Information Technology, or a related field.
  • 7–8 years of overall IT experience, including hands-on Data Engineering experience.
  • Strong communication, analytical, and problem-solving skills.

Skills

SQL
SQL Server
AWS
ETL
MLflow
Airflow
Azure
Databricks
Google Cloud
Unity

Similar jobs

Browse related jobs

Apply now