Start searching
Back to Jobs
Recently postedTechnology

Lead Data Engineer

AIT Global, Inc.Cary, NC🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeHybrid
LocationCary, NC, United States
SQLScalaApache SparkMLOpsData PipelineDatabricksLLMPythonTerraformUnityGenerative AIAzurePrompt EngineeringCI/CDAzure DevOpsAzure Data FactoryDelta LakeLangChainLangGraphRAG

Job description

Job Title: Lead Data Engineer
Location: Cary, NC
Mandatory Skills Must-Have:
  • Python, PySpark, Scala and SQL strong production-level data engineering skills.
  • Azure Databricks Delta Lake, Spark performance tuning, Jobs/Workflows and Unity Catalog.
  • Azure data services ADLS Gen2 and Azure Data Factory (ADF).
  • Lakehouse / Medallion Architecture proven experience implementing Bronze, Silver and Gold layers at enterprise scale.
  • Data Pipeline Engineering metadata-driven ingestion frameworks, data transformation, data quality and large-scale pipeline optimization.
  • Production AI/LLM Experience hands-on experience designing and deploying LLM-based solutions, including RAG, embeddings, retrieval and prompt engineering.
  • LLM Frameworks experience with LangChain, LlamaIndex or LangGraph, plus Azure OpenAI, OpenAI or Databricks Model Serving.
  • Azure DevOps / CI-CD experience automating data pipeline deployments; Terraform and Databricks Asset Bundles are highly relevant.
  • Technical Leadership ability to design solutions, write production code, review code and guide data engineering teams.
  • Financial services or insurance data, especially financial reconciliation, general ledger or accounting data.
  • AI-driven data quality, anomaly detection and automated reconciliation.
  • Knowledge graphs, ontologies, semantic layers or enterprise Text-to-SQL solutions.
  • Azure security, data governance, PII protection and fine-grained access controls.
  • MLOps/LLMOps, model evaluation and production AI monitoring.
Important: This is a hands-on Lead Data Engineer position, not a purely managerial or coordination role. Candidates must demonstrate recent production coding experience.

Similar jobs

Apply