AI Data Engineer - Agentic Data Lineage (Snowflake Cortex)

Job Title: AI Data Engineer – Agentic Data Lineage (Snowflake Cortex)
Location: New York City, NY (Hybrid – 3 days onsite per week)

Job Summary:

We are seeking an experienced AI Data Engineer – Agentic Data Lineage to design and build next-generation AI-powered data lineage and metadata intelligence solutions. This role combines expertise in modern data engineering, large language models (LLMs), agentic AI workflows, and cloud data platforms to automate lineage discovery, metadata enrichment, impact analysis, and governance.

The ideal candidate has strong hands-on experience with Snowflake Cortex AI, PySpark, Python, LLMs, and modern data engineering frameworks. Experience building AI agents capable of reasoning across enterprise metadata and data pipelines is highly desirable. 

Key Responsibilities:

• Design and develop AI-powered data lineage solutions using Agentic AI architectures.
• Build intelligent agents that automatically discover, analyze, and document data lineage across enterprise platforms.
• Develop scalable ETL/ELT pipelines using PySpark and Python.
• Utilize Snowflake Cortex AI capabilities for semantic search, document intelligence, summarization, embeddings, and AI-assisted metadata processing.
• Integrate LLMs (OpenAI, Claude, Llama, Gemini, or similar) into enterprise data engineering workflows.
• Develop Retrieval-Augmented Generation (RAG) solutions for metadata discovery and data catalog search.
• Build metadata extraction pipelines from SQL, Spark jobs, stored procedures, Airflow, dbt, and BI tools.
• Automate impact analysis, dependency mapping, and data governance workflows using AI agents.
• Design vector databases and embedding pipelines for enterprise metadata search.
• Develop APIs and microservices supporting AI-driven lineage services.
• Optimize large-scale Spark workloads for performance and cost efficiency.
• Collaborate with Data Governance, Data Engineering, Analytics, and Architecture teams.
• Implement CI/CD, monitoring, observability, and automated testing for AI and data engineering solutions.
• Ensure compliance with enterprise security, governance, and data privacy standards.

Required Qualifications:

• Bachelor''s or Master''s degree in Computer Science, Data Engineering, Artificial Intelligence, or a related field.
• 5+ years of experience in Data Engineering.
• 2+ years of experience working with Generative AI and LLM-based applications.
• Strong Python programming skills.
• Hands-on expertise with PySpark and Apache Spark.
• Experience with Snowflake Data Cloud.
• Experience using Snowflake Cortex AI capabilities.
• Strong SQL development skills.
• Experience building scalable data pipelines.
• Knowledge of enterprise metadata management and data lineage concepts.
• Experience integrating LLM APIs into enterprise applications.
• Familiarity with vector databases and embedding models.
• Experience with REST APIs and microservices.
• Understanding of distributed data processing architectures.

Preferred Qualifications:

• Experience with Agentic AI frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, or similar.
• Experience with LangChain or LlamaIndex.
• Knowledge of Retrieval-Augmented Generation (RAG).
• Experience with knowledge graphs and graph databases (Neo4j, Amazon Neptune, or similar).
• Experience with Apache Airflow or Prefect.
• Experience with dbt.
• Experience with Kafka or event-driven architectures.
• Familiarity with Databricks.
• Experience with enterprise data governance tools such as Collibra, Alation, Microsoft Purview, Informatica EDC, or Atlan.
• Experience deploying AI applications on AWS, Azure, or Google Cloud Platform.
• Familiarity with Docker, Kubernetes, and Terraform.

Technical Skills:

Programming:

• Python
• PySpark
• SQL

AI & Machine Learning:

• Large Language Models (LLMs)
• Agentic AI
• RAG
• Prompt Engineering
• Embeddings
• Vector Search
• AI Agents
• NLP

Snowflake:

• Snowflake Cortex AI
• Snowpark
• Snowflake SQL
• Snowpipe
• Tasks & Streams

Data Engineering:

• Spark
• ETL/ELT
• Data Pipelines
• Data Warehousing
• Metadata Management
• Data Lineage
• Data Catalog
• Data Governance

Frameworks:

• LangChain
• LangGraph
• LlamaIndex
• CrewAI
• AutoGen
• Semantic Kernel

Cloud & DevOps:

• AWS / Azure / Google Cloud Platform
• Docker
• Kubernetes
• GitHub Actions
• Jenkins
• CI/CD

Nice to Have:

• Experience building autonomous AI agents for enterprise data management.
• Experience with graph-based lineage visualization.
• Knowledge of Microsoft Fabric or Databricks Unity Catalog.
• Exposure to Model Context Protocol (MCP).
• Experience implementing AI governance and responsible AI practices.