Ab Initio Engineer- Azure Migration

Role Overview

We are seeking an experienced Ab Initio expert to lead and support the migration of legacy Ab Initio workloads to Azure Databricks. The candidate will serve as the onshore technical subject-matter expert, bridging business stakeholders and offshore delivery teams while ensuring data accuracy and delivery quality throughout the migration.

Scope of Engagement

• Act as the key Ab Initio SME for the migration of Ab Initio graphs and plans to Azure Databricks ETL jobs, covering assessment, conversion, validation, and cutover.

• Translate Ab Initio logic into Databricks (Spark/PySpark/SQL) equivalents.

• Act as the onshore SME, partnering with business teams and translating requirements into clear specs for offshore delivery.

• Provide guidance, knowledge transfer, and data debugging support to offshore engineers.

• Support data validation during migration and help resolve source-side Ab Initio ETL execution issues.

• Flexible working hours to ensure effective overlap with offshore teams across time zones.

Key Responsibilities

• Analyze and interpret existing Ab Initio graphs, plans, and components to document business and transformation logic.

• Translate complex Ab Initio functions (transform, rollup, join, partition/departition, lookup, normalize/denormalize, etc.) into target Databricks ETL designs.

• Partner with business teams to capture requirements and translate them into actionable specifications for offshore development teams.

• Provide technical guidance and knowledge transfer to offshore engineers executing the conversion.

• Lead data debugging, reconciliation, and root-cause analysis to resolve discrepancies between source (Ab Initio) and target (Databricks) outputs.

• Support data validation and reconciliation throughout the migration to ensure source-to-target parity.

• Troubleshoot and resolve source-side Ab Initio ETL execution issues to keep migration and validation on track.

• Define reusable conversion patterns, templates, and best practices to accelerate delivery.

• Support SIT/UAT cycles, validate data parity, and assist with performance tuning on Databricks.

Required Skills & Experience

• Strong hands-on expertise with Ab Initio (GDE, Co>Operating System, EME, Conduct>It plans, Continuous Flows).

• Deep understanding of Ab Initio graph design, components, and DML/transform functions.

• Working knowledge of Azure Databricks, Apache Spark, PySpark, and SQL.

• Proven experience in legacy ETL modernization or migration programs.

• Strong data analysis, validation, and debugging skills with the ability to trace logic across complex pipelines.

• Excellent communication skills to work across business stakeholders and distributed offshore teams.

Preferred Qualifications

• Overall 5 to 10 years of experience.

• Experience migrating Ab Initio (or comparable legacy ETL such as Informatica/Teradata) to Databricks or other cloud lakehouse platforms.

• Familiarity with Azure data services (ADLS, ADF, Azure SQL) and CI/CD for data pipelines.

• Domain exposure to healthcare, banking, or other regulated, data-intensive industries.

• Experience leading or mentoring offshore delivery teams.