Lead Data Engineer

Introduction:

This senior hands-on leadership role is for a centralized, AI-first enterprise Data Hub for a global insurance and financial services client on Azure Databricks. The Lead Data Engineer will be responsible for owning the end-to-end technical design of the data and AI layers, building reference implementations for the engineering team, and shipping production-grade Python, Scala, and PySpark code weekly.

Responsibilities:

• Data platform architecture and engineering, including lakehouse architecture design
• Developing metadata-driven ingestion frameworks for various data sources
• Building and maintaining PySpark and Scala Spark jobs, coding standards, and testing
• Implementing CI/CD processes for Databricks and ADF in Azure DevOps
• AI-augmented ingestion and canonical mapping
• Driving AI-driven data quality and anomaly detection
• Creating semantic layers and knowledge graphs with security measures
• Providing governance and leadership, including mentoring engineers

Requirements:

Must-Have Skills & Experience:

• Expert-level Python, Scala, and PySpark skills
• Strong SQL and data modeling abilities
• Proficiency in Databricks, Azure data stack, and LLM-based systems
• Evaluation discipline and hands-on experience with LangChain or LlamaIndex
• Metadata-driven frameworks knowledge
• 12-18 years of total experience in data engineering / data platform delivery
• Proven enterprise-scale delivery of a medallion / lakehouse architecture
• Azure security and governance expertise
• Experience with CI/CD and IaC tools
• Clear technical writing and presentation skills

Strongly Preferred:

• Knowledge of knowledge graphs and ontologies
• Experience with ML-based anomaly detection and financial services domain
• LLMOps / MLOps knowledge
• Relevant certifications and exposure to specific tools