W2 ONLY -Data Engineer ( Databricks, Scala , PYSPARK)

Experienced Senior Data Engineer to work on a focused migration project — converting existing Scala-based Spark pipelines to PySpark and transitioning job workloads from classic Databricks clusters to Databricks Serverless compute. The ideal candidate has hands-on experience with both Scala and PySpark, deep knowledge of Azure Databricks, and a strong understanding of Serverless compute constraints and best practices.

Key Responsibilities

• Review and analyze existing Scala Spark jobs — identify dependencies, performance bottlenecks, and Serverless compatibility issues.
• Convert Scala pipelines to PySpark using AI-assisted tools (GitHub Copilot / Databricks Genie) with thorough manual review and validation.
• Ensure migrated PySpark code matches original business logic, data quality standards, and pipeline performance.
• Resolve Serverless incompatibilities — custom JARs, JDBC drivers, DBFS paths, streaming triggers, and unsupported Spark configurations.
• Migrate Databricks job cluster definitions to Serverless compute with updated library and parameter configurations.
• Perform data validation and runtime benchmarking — row counts, schema checks, and side-by-side output comparison.
• Document migration changes, known issues, and post-migration runbook for production cutover.