Back to Jobs
Recently postedRemoteTechnology

Senior Data Engineer- SQL Server

eSolutionsFirst, LLCPhiladelphia, PA🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeRemote
LocationPhiladelphia, PA, United States
SQLT-SQLGitApache SparkDatabricksUnityAzureSQL ServerDelta Lake

Job description

Senior Data Engineer- SQL Server to Databricks Migration

Job Type: Contract (6-12 Months)- Possible extension

Location: Philadelphia, PA / Vienna, VA - Hybrid / Open to remote job

Job description:

ü Job summary

ü We are seeking a senior data engineer with deep Microsoft SQL Server experience to assess and convert a legacy warehouse to Azure Databricks.

ü You will read complex T-SQL stored procedures and SSIS packages, document the business rules they contain, rebuild them as Databricks pipelines, and reconcile the results before the legacy platform is retired.

Key responsibilities

ü Assess T-SQL stored procedures, SSIS packages and scheduled jobs for complexity, dependencies, and embedded business rules.

ü Recommend, for each object, whether to retire, convert or redesign it.

ü Convert legacy logic to Databricks SQL and PySpark pipelines using agreed patterns and conversion tooling.

ü Review and correct automatically converted code.

ü Build and run reconciliation between SQL Server and Databricks and resolve differences with business analysts.

ü Migrate job schedules and dependencies to Databricks workflows.

ü Support cutover and the evidence needed to decommission each legacy component.

ü Document data flows, mappings and design decisions in the client's repositories.

Required qualifications

ü 8+ years in data engineering or database development, including 5+ years of hands-on T-SQL and SSIS.

ü 2+ years building pipelines on Databricks, or an equivalent Spark-based lakehouse platform, with Delta Lake.

ü At least one completed migration from SQL Server or another warehouse to a cloud data platform.

ü Strong SQL performance tuning and the ability to reverse-engineer undocumented code.

ü Experience with data reconciliation and testing at scale.

ü Git-based development with pull requests and automated deployment.

Preferred qualifications

ü Databricks Lakeflow pipelines and jobs, and Unity Catalog.

ü Code-conversion tools such as Databricks Lakebridge.

ü Databricks Data Engineer Associate or Professional certification.

ü Financial services or wealth management data.

ü Experience with enterprise schedulers such as SQL Agent or ActiveBatch.

Will your resume get past this employer's screening? Check your resume against this job, free

Similar jobs