Haystack
← Back to Jobs
Technology
CL

W2 ONLY -Data Engineer ( Databricks, Scala , PYSPARK)

cloudingest incDallas, TX🇺🇸United StatesPosted 14 Sept 2026

Why This Role Stands Out

This hybrid role offers a fantastic opportunity to leverage your Databricks, Scala, and PySpark expertise on a critical migration project, enhancing your skills with cutting-edge tools. You'll thrive here if you're a mid-senior data engineer eager to drive significant impact within a reputable company and contribute to innovative cloud solutions. Apply now to be part of this exciting transformation and advance your career!

Quick Overview

Seniority
Mid Senior
Work mode
Hybrid
Location
Dallas, TX, United States
Posted
2 days ago
ScalaAzureDatabricks

Job Description

Experienced Senior Data Engineer to work on a focused migration project — converting existing Scala-based Spark pipelines to PySpark and transitioning job workloads from classic Databricks clusters to Databricks Serverless compute. The ideal candidate has hands-on experience with both Scala and PySpark, deep knowledge of Azure Databricks, and a strong understanding of Serverless compute constraints and best practices.

Key Responsibilities

  • Review and analyze existing Scala Spark jobs — identify dependencies, performance bottlenecks, and Serverless compatibility issues.
  • Convert Scala pipelines to PySpark using AI-assisted tools (GitHub Copilot / Databricks Genie) with thorough manual review and validation.
  • Ensure migrated PySpark code matches original business logic, data quality standards, and pipeline performance.
  • Resolve Serverless incompatibilities — custom JARs, JDBC drivers, DBFS paths, streaming triggers, and unsupported Spark configurations.
  • Migrate Databricks job cluster definitions to Serverless compute with updated library and parameter configurations.
  • Perform data validation and runtime benchmarking — row counts, schema checks, and side-by-side output comparison.
  • Document migration changes, known issues, and post-migration runbook for production cutover.

Similar jobs

Browse related jobs

Apply now