Lead Data Engineer

Job Description:
We are seeking a hands-on Lead Data Engineer to design, develop, and lead enterprise-scale Data & AI platforms using Azure Databricks. The ideal candidate will have strong expertise in data engineering, cloud architecture, Spark, and production-grade GenAI/LLM solutions.
Key Responsibilities

• Design and implement Lakehouse / Medallion Architecture using Bronze, Silver, and Gold layers.
• Develop scalable data pipelines using Python, Scala, PySpark, SQL, and Spark.
• Build data ingestion frameworks using Azure Data Factory, ADLS Gen2, Kafka, and Event Hubs.
• Develop and optimize Databricks / Delta Lake workloads.
• Implement CI/CD using Azure DevOps and Terraform.
• Build AI-powered data quality, anomaly detection, and reconciliation solutions.
• Design and implement RAG, LLM, and Agentic AI solutions.
• Work with LangChain, LangGraph, LlamaIndex, Azure OpenAI, or Databricks Model Serving.
• Implement data governance using Unity Catalog, RBAC, PII protection, and Azure security controls.
• Lead technical design reviews, mentor engineers, and establish development standards.

Required Skills

• 12 18 years of Data Engineering experience.
• Expert-level Python, PySpark, and Scala.
• Strong Azure Databricks, Delta Lake, Unity Catalog experience.
• Hands-on ADLS Gen2 and Azure Data Factory.
• Strong SQL and data modeling skills.
• Experience with Kafka / Event Hubs / Spark Structured Streaming.
• Strong Azure DevOps, Terraform, and CI/CD experience.
• 3+ years of production GenAI/LLM experience.
• Experience with RAG, Agentic AI, embeddings, vector search, and prompt engineering.
• Strong architecture, troubleshooting, and technical leadership skills.