AI Databricks Engineer

Mandatory skills

• Data Engineering Design and develop scalable ETL/ELT pipelines using Databricks, Spark, and SQL. Build data ingestion frameworks for structured and unstructured documents. Develop data quality, reconciliation, and validation frameworks.
• Design medallion architecture (Bronze, Silver, Gold) for document processing workflows. Implement metadata management and lineage tracking. Gen AI & Agentic AI Build Intelligent Document Processing (IDP) solutions using LLMs.
• Develop AI agents to perform: Contract clause extraction Invoice validation Payment term identification Business rule evaluation Exception handling workflows Implement RAG (Retrieval Augmented Generation) solutions. Design multi-agent workflows using Microsoft Agentic Framework, LangChain, LangGraph, CrewAI, Semantic Kernel or similar.
• Develop prompt engineering and evaluation frameworks. Build human-in-the-loop review workflows. OCR & Document Intelligence Integrate OCR platforms such as: Azure AI Document Intelligence Azure OCR AWS Textract Google Document AI ABBYY Extract: Vendor information Invoice details PO details Payment schedules Contract clauses Pricing schedules Improve extraction accuracy through AI-assisted validation.
• Stakeholder Management Work closely with business SMEs and invoicing teams. Conduct requirement workshops. Present solution architecture and implementation progress to stakeholders.

Technical Skills Databricks Apache Spark (PySpark) SQL Python Azure Data Engineering Data Modeling Gen AI / LLM Integration Agentic AI Frameworks RAG Architecture OCR / Document Intelligence REST API Development AI Technologies Azure OpenAI GPT Models Semantic Kernel LangChain LangGraph CrewAI Vector Databases Cloud Platforms Azure (Preferred) AWS Google Cloud Platform

Preferred Skills Contract lifecycle management domain knowledge Accounts Payable / Invoicing experience Knowledge Graph integration experience Healthcare or Financial Services domain experience