Quick overview
Job description
Databricks Pyspark Engineer
Plano, TX
Must have hands on recent coding experience on databricks
Role Summary
We are seeking a to drive the end to end successful migration of enterprise data integration workloads from Informatica to Azure Databricks and to provide thought leadership for the Databricks platform within the supply chain ecosystem. This role combines hands on technical leadership, architecture guidance, delivery ownership, stakeholder alignment, and platform enablement. The ideal candidate will lead modernization efforts that improve scalability, reliability, performance, and business value across supply chain data products and analytics capabilities.
Key Responsibilities
· Lead the end to end migration of data pipelines, workflows, and integration patterns from Informatica to Azure Databricks.
· Assess current state Informatica assets and define target state architecture, migration strategy, and phased execution plans. The current environment is based on Informatica PowerCenter 10.4.1 running batch ETL workloads, with Oracle as both the source and target database. The target state platform is Azure Databricks (Runtime 14.4 LTS) with Unity Catalog, Delta Live Tables (DLT), and ADLS Gen2.
· Drive modernization of ETL patterns into scalable ELT and data engineering approaches aligned with Azure Databricks best practices.
· Analyze and interpret complex Informatica PowerCenter mappings, including transformations, lookups, joins, filters, routers, and expression logic, and build equivalent PySpark based pipelines in Azure Databricks.
· Assess Oracle database level stored procedures, functions, and packages that are invoked by or tightly coupled with Informatica workflows, and determine whether to migrate, refactor, or recreate that logic within Databricks using PySpark, Python, or Spark SQL.
· Provide technical leadership across solution design, engineering standards, code quality, testing strategy, deployment, and production readiness.
· Partner with business, product, architecture, infrastructure, security, and operations stakeholders to align migration outcomes with enterprise priorities.
· Establish migration success criteria covering data quality, reconciliation, performance, resilience, observability, and business continuity.
· Identify delivery risks, dependencies, and technical constraints early, and drive mitigation plans through execution.
· Guide cutover, stabilization, and post migration optimization to ensure sustainable platform adoption and operational excellence.
· Mentor engineers and technical teams on Databricks design patterns, reusable frameworks, and modern data engineering practices
Similar jobs
- LL
Lead Protection Relay Testing & Commissioning Engineer
L. L. Blue Engineering · Atlanta, GA · Hybrid · Posted 2 Nov 2026 🇺🇸
PLC ProgrammingSCADAEngineeringSalary not listed - OR

Lead Principal Supply Chain EngineerRecently posted Auto Apply
Oracle · Houston · Hybrid · Posted 17 Nov 2026 🇺🇸
SourcingOracleAssembly+4Engineering$102.3k - $209.5k/yr - OR

Principal Electrical Engineer, Server Hardware DesignRecently posted Auto Apply
Oracle · Burlington · Hybrid · Posted 16 Nov 2026 🇺🇸
OnboardingOracleFPGA+4Engineering$114.6k - $234.6k/yr - OR

Marketing Technology EngineerRecently posted Auto Apply
Oracle · United States · Hybrid · Posted 12 Nov 2026 🇺🇸
OnboardingOraclePL/SQL+8Engineering$74.1k - $148.3k/yr - CF
Staff Fuel Cycle and Blanket Chemical EngineerRecently posted Auto Apply
Cfsenergy · Devens · Hybrid · Posted 17 Nov 2026 🇺🇸
ComplianceConfluenceProcurement+2Engineering$152k - $230k/yr - VI

PKI Engineer with Venafi Certification- RemoteRecently posted
Visionary Innovative Technology Solutions · United States · Remote · Posted 17 Nov 2026 🇺🇸
EngineeringSalary not listed - BS

Senior Principal II DevSecOps EngineerRecently posted
BAE Systems · Nashua, NH · Hybrid · Posted 17 Nov 2026 🇺🇸
AgileData AnalysisEngineering$149.6k - $254.3k/yr - AI

Systems Safety Engineer
ANDURIL INDUSTRIES · Seattle, WA · Hybrid · Posted 2 Nov 2026 🇺🇸
ANSYSJiraEngineering$126k - $167k/yr - AI

Systems Safety Engineer
ANDURIL INDUSTRIES · Fairfax, VA · Hybrid · Posted 2 Nov 2026 🇺🇸
ANSYSJiraEngineering$146k - $194k/yr - AI

Systems Safety Engineer
ANDURIL INDUSTRIES · Arlington, VA · Hybrid · Posted 2 Nov 2026 🇺🇸
ANSYSJiraEngineering$146k - $194k/yr - AI

Systems Safety Engineer
ANDURIL INDUSTRIES · Kensington, MD · Hybrid · Posted 28 Oct 2026 🇺🇸
ANSYSJiraEngineering$146k - $194k/yr - AI

Systems Safety Engineer
ANDURIL INDUSTRIES · Lorton, VA · Hybrid · Posted 28 Oct 2026 🇺🇸
ANSYSJiraEngineering$146k - $194k/yr
