Haystack
← Back to Jobs
Engineering

Lead Databricks Engineer with Python (Need Only local to MN)

Sovereign TechnologiesEagan, MN🇺🇸United StatesPosted 22 Jul 2026

Why This Role Stands Out

You can lead a critical healthcare analytics pipeline modernization, migrating legacy SAS workflows to Python/PySpark on Databricks for a reputable company. This hybrid role offers a fantastic opportunity for a mid-senior engineer with strong Python and Databricks skills to drive significant impact and grow their expertise in a collaborative environment. Explore this contract-to-hire position to leverage your skills and advance your career.

Quick Overview

Work Type
On Site
Level
Mid Senior

Job Description

Title: Senior / Lead Data Engineer

Location: Eagan, MN (Hybrid – 2 days/week onsite)
Duration: 3–6 Months Contract-to-Hire
Start Date: ASAP

Job Summary

We are seeking a Senior/Lead Data Engineer to modernize and own enterprise data pipelines by migrating legacy SAS-based workflows to Python/PySpark on Databricks. The engineer will partner with SMEs during the transition period and eventually take ownership of a critical healthcare analytics pipeline supporting HEDIS reporting.

Key Responsibilities

  • Migrate legacy SAS pipelines to Python/PySpark on Databricks.
  • Design, build, and maintain scalable ETL/ELT pipelines.
  • Develop distributed data processing solutions using Databricks.
  • Create and optimize complex SQL queries.
  • Schedule, automate, and monitor data pipelines.
  • Work with AWS services including S3, Lambda, Glue, and EC2.
  • Manage Databricks notebooks, workflows, and clusters.
  • Collaborate with business stakeholders and SMEs to transition pipeline ownership.
  • Follow Agile methodologies and Git-based development practices.

Required Skills

  • 8+ years of Data Engineering experience.
  • Strong Python programming and scripting skills.
  • Hands-on PySpark development.
  • Extensive Databricks experience (Notebooks, Workflows, Cluster Management).
  • Experience processing large datasets in distributed environments.
  • Strong SQL skills.
  • AWS experience with:
    • S3
    • Glue
    • Lambda
    • EC2
  • Experience building and automating ETL/ELT pipelines.
  • Git/version control experience.
  • Agile/Scrum experience.

Preferred Skills

  • Experience with SAS-to-Python migration.
  • AI/Automation experience.
  • Healthcare or HEDIS domain experience.
  • Lead/Principal Data Engineering experience.

Must-Have Skills

  • Python
  • PySpark
  • Databricks
  • SQL
  • AWS (S3, Glue, Lambda, EC2)
  • ETL/ELT Pipeline Development
  • Distributed Data Processing
  • Git
  • Agile/Scrum

Nice-to-Have Skills

  • SAS Modernization
  • AI/Generative AI
  • Healthcare/HEDIS
  • Airflow or other workflow orchestration tools
  • Leadership/Pipeline Ownership experience

Skills

Scrum
Agile

Similar jobs

Browse related jobs

Apply now