Quick Overview
Seniority
Mid Senior
Work mode
Remote
Location
United States
Posted
18 hours ago
ETLDatabricksLLMPython
Job Description
Job Description
AI Architect - AI & Automation
Work Location: Remote
Key Responsibilities:
- Own the technical roadmap for AI-driven data modeling, documentation generation, data quality rule generation, and transformation code generation.
- Design agentic workflow architecture using frameworks such as LangGraph, CrewAI, or AutoGen.
- Ensure all AI workflows operate within the client's AI governance and approval framework.
- Lead a small delivery team (AI Tech Lead, AI Developer) through discovery, prioritization, and iterative delivery of automation solutions.
- Act as the primary technical point of contact with client platform and data engineering stakeholders.
- Required Qualifications 10+ years in data engineering or applied AI/ML, including 2+ years leading applied-AI delivery teams.
- Proficiency on Databricks Platform Experience designing agentic or multi-step AI workflows (LangGraph, CrewAI, AutoGen, or equivalent).
- Experience building LLM-powered applications, including prompt engineering, context engineering, and Retrieval-Augmented Generation (RAG).
- Experience integrating large language models into data engineering pipelines within a governed enterprise environment.
- Experience with LLM evaluation and human-in-the-loop validation workflows to ensure accuracy and reliability of AI-generated outputs; working knowledge of guardrails and responsible-AI practices.
- Experience building reusable, modular GenAI components (e.g., plugins) using structured multi-agent patterns to enable repeatable automation.
- Experience deploying, versioning, monitoring, and evaluating LLM solutions in production, including prompt/version management and feedback-loop mechanisms.
- Strong understanding of ETL/ELT patterns, schema analysis, source-to-target mapping, data quality, data profiling and data modeling.
- Experience in designing reusable framework integration patterns with Databricks Python, PySpark/Declarative pipelines
- Understanding of Data Engineering frameworks such as Data Quality, Data Modelling, Metadata Enrichment, Sensitive data handling, STTM, Modular and testable Python or PySpark/Declarative pipeline code generation.
- Experience in designing performance and cost optimized data engineering frameworks Client-facing consulting or delivery experience.
- Experience leveraging Agentic coding platforms (for example, GitHub Copilot or Claude Code) effectively.
Preferred Qualifications:
Experience with Databricks AI/ML tooling.
Background in data quality automation or catalog/metadata generation.
Similar jobs
- ND
AI Scientist - PhD Intern
NewNtt Data Aivista
San Francisco Bay Area🇺🇸$12.5k - $14.5k/moHybrid12 hours agoMachine LearningPython - SN
AI Solutions Specialist
NewSnowflake
United States🇺🇸Remote5 hours agoMachine LearningSnowflakeContinuous Improvement - BA
Senior AI Engineer, Agentic Data Enrichment
NewBaselayer
San Francisco🇺🇸Hybrid7 hours agoCDNData PipelineLLM+2Technology - TE
AI Developer Security Scanning
NewTekShapers
United States🇺🇸Hybrid18 hours agoAWSSplunkAgile+3Technology - IN
AWS AI ENGINEER
NewInvolgix
San Antonio, TX🇺🇸On-site18 hours agoDynamoDBAPI GatewayAWS+2Technology - MA
AI Engineer with Glean AI Experience
NewModern Agile Technologies, LLC
San Francisco, CA🇺🇸Hybrid18 hours agoMachine LearningNLPAgile+2Technology