Haystack
← Back to Jobs
Technology

Mid/Senior Data Engineer – AI Data Pipelines

Pegasus Knowledge SolutionsNew York, NY🇺🇸United StatesPosted 14 Jul 2026

Why This Role Stands Out

This hybrid role offers a fantastic opportunity to build cutting-edge AI data pipelines and shape the future of AI agents, perfect for a skilled Data Engineer eager to make a significant impact. You'll gain invaluable experience with vector databases and large-scale data processing, all within a collaborative environment. Apply now to leverage your expertise and grow your career!

Quick Overview

Work Type
Hybrid
Level
Mid Senior

Job Description

<>Mid/Senior Data Engineer – AI Data Pipelines  <>NYC, NY <>Contract 6+ Months with possible extension

We’re hiring a strong Data Engineer to turn messy customer support data into clean, AI‑ready intelligence. You’ll build high‑scale pipelines, vector DBs, and reliable data systems that power our next‑gen AI agents.

Role Highlights

  • Clean, process & structure huge volumes of unstructured support data (Salesforce, Sprinklr, Bliss, JIRA).
  • Build AI memory systems using vector databases (Pinecone, Milvus, Weaviate, pgvector).
  • Create a central metrics hub for analytics & AI insights.
  • Own high‑reliability pipelines with monitoring & zero‑downtime standards.
  • Work closely with product/ops and push for better data logging at the source.

Tech Stack

  • Python, PySpark
  • Spark, Kafka, Flink, Hadoop, Hudi, Presto, Pinot
  • Vector DBs (Pinecone/MilvWeaviate/pgvector)
  • Cloud data warehouses
  • AI tools (Claude, Codex, etc.) for automation & parsing

Ideal Candidate

  • 5+ years in data engineering with big data systems
  • Strong Python + PySpark
  • Experience building AI‑ready data layers
  • Proactive, independent, business‑focused
  • Comfortable working in ambiguity and driving clarity
  • Honest communicator who flags risks early

Skills

Flink
Hadoop
Jira
Kafka
Python

Similar jobs

Apply now