Lead Data Engineer

Mandatory skills

• 10+ years of Data Engineering experience with strong focus on modern cloud data platforms. Hands-on experience with Databricks, PySpark, SQL, and Python.

• Strong understanding of ETL/ELT design patterns, data integration, and data warehousing concepts. Experience with relational databases, data modeling, partitioning, indexing, and query optimization.

• Expertise in building and maintaining large-scale data pipelines and batch processing solutions. Experience working with cloud platforms such as Azure, AWS, or Google Cloud Platform. Strong problem-solving, debugging, and analytical skills. Proven experience architecting and delivering systems using agentic IDEs

• Ability to: Define architectural intent that agents can follow • Break features into agent executable tasks • Govern AI autonomy (guardrails, permissions, reviews) • Integrate agentic workflows into CI/CD pipelines

• Experience supervising AI agents across: Multi service systems • Legacy modernization • Large codebases / monorepos

• Strong understanding of: Security implications of autonomous code execution • Compliance, auditability, and traceability • AI assisted SDLC operating models

• Core Responsibility: • Guide effective use of agentic IDEs for complex, multi-module or cross-service changes • Establish review practices and quality checks for AI-generated code • Mentor team members on balancing autonomy, correctness, and maintainability in AI-assisted development

• Design system architectures that support AI-augmented and agentic development workflows • Define guardrails, standards, and governance for the use of autonomous coding agents • Evaluate impact of agentic IDEs on SDLC, CI/CD pipelines, security posture, and technical debt.

• We are currently looking for candidates who are authorized to work in the United States without requiring current or future visa sponsorship.