Azure Databricks & Agentic AI Architect

Job Title: Azure Databricks & Agentic AI Architect

Location: - Chicago, IL(Hybrid- 3 days/week in office)

Employment Type: Long term contract

Job Description:-

Ideal Candidate Profile

A forward-looking architect with 15+ years of Data & Analytics experience, strong Azure Databricks expertise, and hands-on experience building Agentic AI platforms, AI-powered SDLC frameworks, and autonomous data engineering ecosystems. The candidate should be comfortable leading enterprise-scale AI transformation initiatives and engaging CXO stakeholders on AI strategy and business value.

Mandatory skills

• Azure, DataBricks, Agentic AI, ETL, SQL, Data Engineering

We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.

The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.

Key Responsibilities

Agentic Data Engineering Leadership

• Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.
• Define autonomous workflows using AI Agents for:
• Data ingestion
• Data mapping
• Schema evolution
• Data quality remediation
• Metadata Enrichment
• Pipeline optimization
• Root cause analysis
• Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.

AI-Driven Data Lifecycle (AI-DLC)

• Lead architecture for AI-enabled Data Development Lifecycle across:
• Requirement analysis
• Data modeling
• Pipeline generation
• Automated testing
• Code review
• Documentation
• Deployment
• Monitoring
• Implement AI copilots to accelerate developer productivity.
• Enable automated lineage creation and intelligent impact analysis.

Lakehouse & Data Platform Architecture

• Design scalable Lakehouse platforms using:
• Azure Databricks
• Delta Lake
• Unity Catalog
• ADLS Gen2
• Databricks Workflows
• Delta Live Tables

Enterprise GenAI Integration

• Architect RAG-based solutions using enterprise data assets.
• Design agent orchestration frameworks using:
• Azure OpenAI
• LangGraph
• Semantic Kernel
• AutoGen
• MCP-enabled architectures
• Build domain-specific AI agents supporting Data Engineering and Analytics teams.

AI Governance & Responsible AI

• Define guardrails for enterprise GenAI adoption.
• Implement:
• Prompt governance
• Observability
• Cost monitoring
• Auditability
• Explainability
• Security controls
• Establish governance models for autonomous AI agents.

AI-Powered Platform Optimization

• Design self-healing data pipelines.
• Implement AI-driven:
• Incident triage
• Failure prediction
• Capacity planning
• Cost optimization
• SLA monitoring
• Enable intelligent workload placement and model routing.

Technical Skills

• Data Platform
• Azure Databricks
• Delta Lake
• Unity Catalog
• Azure Data Factory

AI & Agentic Frameworks

• Azure OpenAI
• Knowledge Graph
• RAG Architecture
• LangChain
• LangGraph
• MCP Protocol
• Vector Databases
• AI Agent Orchestration

Data Engineering

• PySpark
• Spark SQL
• Python
• SQL
• ELT/ETL Modernization

DevOps & AI-DLC

• Azure DevOps
• GitHub Actions
• CI/CD
• MLOps
• LLMOps
• Evaluation Frameworks
• AI Testing Frameworks

Leadership Expectations

• Drive AI-First Data Engineering transformation.
• Define enterprise patterns, accelerators, and reusable AI agents.
• Mentor architects, data engineers, and AI engineers.
• Lead executive conversations on AI adoption, ROI, and transformation roadmaps.

Preferred Certifications

• Databricks Certified Data Engineer Professional
• Databricks Certified Solution Architect
• Microsoft Azure Solution Architect (AZ-305)
• Azure Data Engineer (DP-203)
• Microsoft Applied Skills - Azure OpenAI
• Generative AI / Agentic AI Certifications

Success Metrics

• 30-50% Data Engineering productivity improvement.
• Reduction in manual pipeline development effort.
• Improved data quality and governance compliance.
• Measurable ROI from Agentic AI adoption.
• Expansion of reusable AI agents across programs.