Senior Databricks & AI Engineer/Lead _ AY

We are seeking a visionary Senior Databricks & AI Engineer to lead the evolution of our modern data platform and AI roadmap.
You will be responsible for designing scalable Spark-based architectures.
Key Responsibilities
Architectural Leadership: Design and govern end-to-end data architectures on the Databricks Data Intelligence Platform.
AI Innovation: Lead the design of emerging AI workflows, specifically focusing on Agentic AI (multi-agent systems), LLM orchestration (LangChain/LlamaIndex), and RAG patterns tailored for insurance policy and regulatory documents.
Data Engineering Excellence: Standardise data engineering templates (Medallion architecture) and implement robust data modelling (Star/Snowflake, Vault 2.0) to handle structured and unstructured insurance data.
Governance & Security: Ensure all designs comply with industry regulations using Databricks Unity Catalog.

Required Skills & Experience
Experience: 8+ years in Data Engineering/Architecture, with at least 3+ years of hands-on leadership within the Databricks ecosystem.
Core Tech: Expert-level knowledge of the Spark framework (PySpark/Scala), Delta Lake, and Databricks SQL.
AI/ML Expertise: Proven experience building and deploying AI solutions. Deep understanding of the Agentic AI landscape—moving beyond simple chatbots to autonomous, goal-oriented agents.
Data Management: Strong background in data modelling and the creation of reusable ETL/ELT templates and frameworks.