AI Solution Architect

Job Description: AI Architect Agentic AI for Data Engineering (AI-SDLC) Define the vision, solution architecture, and roadmap for embedding AI and agents across the data engineering delivery lifecycle Looking for an experienced AI Architect with a strong data engineering background to lead the design and implementation of AI and agentic AI capabilities across the AI-enabled Software Development Lifecycle (AI-SDLC) for our Data Engineering practice. In this role, you will define the target-state architecture, build a phased adoption roadmap, and deliver reusable AI agents and accelerators that automate and improve every stage of data engineering delivery, from requirements and design through development, testing, deployment, and operations. You will work closely with practice leadership, delivery teams, and clients to improve productivity, quality, and speed-to-value, while ensuring strong governance, security, and responsible AI practices. Key Responsibilities AI Strategy, Solutioning & Roadmap Assess the current maturity of the data engineering practice and identify high-value opportunities to apply AI and agents across the delivery lifecycle. Define the AI-SDLC vision, target-state reference architecture, and a phased implementation roadmap with clear milestones, KPIs, and business outcomes. Build business cases that quantify productivity gains, cost savings, quality improvements, and delivery acceleration. Evaluate and recommend AI platforms, large language models (LLMs), agent frameworks, and developer tools, balancing capability, cost, security, and client fit. Lead proofs of concept and pilots, and scale successful solutions into standard, repeatable practice offerings. Support pre-sales, RFP responses, and client workshops with AI-SDLC solution approaches, architectures, and demos. Agentic AI Across the Data Engineering Lifecycle Requirements and analysis: Design agents that turn business requirements into user stories, source-to-target mappings, and data contracts, and that profile source systems automatically. Design: Enable AI-assisted data modeling, schema design, and architecture documentation, including generation of design artifacts and lineage views. Development: Implement code generation agents for SQL, PySpark, dbt, and ETL/ELT pipelines, including legacy code conversion (for example, Informatica, DataStage, SSIS, or Teradata to modern cloud platforms). Code review and quality: Build agents that review code against standards, recommend performance tuning, and flag security and compliance issues. Testing: Automate test case generation, synthetic test data creation, data reconciliation, and data quality validation. Deployment: Integrate AI into CI/CD and DataOps pipelines for automated release notes, impact analysis, and deployment validation. Operations and support: Design agents for pipeline monitoring, anomaly detection, root cause analysis, self-healing workflows, and incident ticket triage. Documentation and knowledge: Build assistants that generate technical documentation and metadata, and answer questions about the data estate using retrieval-augmented generation (RAG). Design multi-agent orchestration patterns, including tool use, memory, planning, human-in-the-loop checkpoints, and agent-to-agent communication. Data Engineering & Platform Architecture Architect AI-ready data platforms on modern cloud technologies such as Databricks, Snowflake, Google BigQuery, Azure Fabric/Synapse, and AWS. Design the metadata, catalog, and lineage foundations (for example, Unity Catalog, Purview, or Dataplex) that AI agents need for context and grounding. Define integration patterns between agents and the data engineering toolchain, including orchestrators, version control, CI/CD, ticketing, and data quality tools, using APIs and Model Context Protocol (MCP) where appropriate. Build reusable accelerators, prompt libraries, agent templates, and reference implementations that delivery teams can adopt quickly. Ensure that solutions are scalable, cost-optimized, and observable, with clear LLMOps and AgentOps practices. Governance, Security & Responsible AI Define AI governance frameworks covering model selection, data privacy, access control, intellectual property, and auditability of AI-generated code. Establish guardrails, evaluation frameworks, and quality gates for agent outputs, including accuracy, hallucination, and security checks. Ensure compliance with client security policies and with regulatory and responsible AI standards. Define and track adoption and value metrics such as productivity improvement, defect reduction, cycle time, and cost per deliverable. Leadership & Change Enablement Act as the AI-SDLC thought leader and trusted advisor for practice leadership, delivery teams, and clients. Lead an AI Center of Excellence (CoE) or community of practice to drive adoption, share best practices, and capture lessons learned. Create training programs and playbooks to upskill data engineers on AI-assisted development and agent usage. Present roadmaps, progress, and value realized to executive and client stakeholders Technical Expertise Area Tools & Technologies Generative AI & LLMs Azure OpenAI, Google Gemini / Vertex AI, AWS Bedrock, Anthropic Claude, open-source LLMs; prompt engineering, fine-tuning, embeddings, RAG Agent Frameworks LangChain, LangGraph, Microsoft Semantic Kernel, AutoGen, CrewAI, Vertex AI Agent Builder, Azure AI Foundry Agent Service, Model Context Protocol (MCP) AI Developer Tools GitHub Copilot, Gemini Code Assist, Amazon Q Developer, Cursor, Databricks Assistant Data Platforms Databricks, Snowflake, BigQuery, Microsoft Fabric, Azure Synapse, AWS Redshift, Delta Lake, Apache Iceberg Data Engineering Apache Spark / PySpark, SQL, dbt, Airflow, Azure Data Factory, Kafka, Informatica, legacy ETL migration Vector & Knowledge Stores Vertex AI Vector Search, Azure AI Search, Pinecone, pgvector, Databricks Vector Search, knowledge graphs DevOps, DataOps & LLMOps Git, Azure DevOps, GitHub Actions, Jenkins, Terraform, Docker, Kubernetes, MLflow, LangSmith, agent observability tools Governance & Quality Unity Catalog, Microsoft Purview, Dataplex, Collibra, Great Expectations, AI guardrails and evaluation frameworks Required Qualifications Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field. 12+ years of IT experience, including 8+ years in data engineering, data warehousing, or data platform architecture. 3+ years of hands-on experience designing and delivering AI, generative AI, or LLM-based solutions in production. Proven experience building agentic AI solutions, including multi-agent orchestration, tool integration, and human-in-the-loop workflows. Deep knowledge of the software and data engineering delivery lifecycle, Agile and DevOps practices, and DataOps. Strong hands-on skills in Python and SQL, plus experience with Spark and at least one major cloud platform (Azure, Google Cloud, or AWS). Track record of defining technology roadmaps and driving adoption across large delivery teams or practices. Excellent communication and storytelling skills, with experience presenting to executive and client stakeholders. Preferred Qualifications Cloud AI or architecture certifications such as Google Cloud Professional Machine Learning Engineer, Microsoft Azure AI Engineer, AWS Machine Learning Specialty, or Databricks Generative AI Engineer. Experience building AI-driven accelerators for legacy ETL and data warehouse migration (for example, Teradata, Informatica, Palantir, or SSIS to Databricks, Snowflake, or BigQuery). Experience establishing an AI Center of Excellence or practice-level AI adoption program in a consulting or services environment. Domain experience in retail, consumer goods, financial services, or healthcare. Familiarity with AI regulations and frameworks such as the EU AI Act, NIST AI Risk Management Framework, and ISO/IEC 42001