Lead Data Engineer (Foundry)

We are seeking a Lead Data Engineer to provide hands-on technical leadership across our Data & Analytics platform, with Palantir Foundry serving as a core component of our data ecosystem.

This role is responsible for the architecture, reliability, scalability, and technical quality of our data products and pipelines. The Lead Data Engineer will work closely with product, functional, analytics, and business teams to translate requirements into durable, production-ready solutions while establishing engineering standards and mentoring other engineers.

The ideal candidate combines strong data engineering fundamentals with deep hands-on experience in Palantir Foundry and is comfortable operating across the full data product lifecycle, from source ingestion and transformation through ontology, applications, monitoring, and production support.

Key Responsibilities

• Lead the design and implementation of scalable data pipelines, data models, integrations, and business-ready data products.
• Own technical architecture and engineering standards across the data platform, with a focus on maintainability, reliability, performance, and scalability.
• Design scalable, layered data architectures that clearly separate source-aligned, transformed, business-ready, and semantic/ontology data while maintaining lineage and governance.
• Develop solutions using Palantir Foundry, including Code Repositories, Pipeline Builder, Ontology, Workshop, Actions/Functions, Data Lineage, and related platform capabilities.
• Build and optimize transformations using Python, PySpark, and SQL, including incremental processing, late-arriving data, reconciliation, and data-quality controls.
• Review existing pipelines and platform architecture, identify risks or technical debt, and define pragmatic, incremental improvements.
• Establish and enforce standards for code review, testing, CI/CD, monitoring, health checks, documentation, and production readiness.
• Ensure data products are governed, traceable, secure, and aligned to approved business definitions and source-of-truth decisions.
• Troubleshoot production issues, perform root-cause analysis, and improve observability and reliability across critical data products.
• Mentor junior and senior engineers through design reviews, code reviews, reusable patterns, and hands-on technical guidance.
• Partner with product and functional teams to evaluate technical feasibility, clarify dependencies, and break larger solutions into executable engineering work.
• Support the design and implementation of Foundry Ontology and Workshop-based applications where appropriate.

Required Qualifications

• Strong hands-on experience with Palantir Foundry in a production environment.
• Advanced experience with Python, PySpark, and SQL.
• Strong understanding of layered data architecture patterns (including medallion or comparable approaches), ETL/ELT, dimensional and semantic modeling, incremental processing, data quality, reconciliation, and lineage.
• Experience designing and supporting production-grade data pipelines and integrations across multiple source systems.
• Experience with Git-based development, CI/CD, automated testing, monitoring, and release management.
• Strong understanding of data governance, access controls, security, and production support practices.
• Demonstrated ability to lead technical design, review engineering work, mentor engineers, and establish repeatable standards.
• Ability to communicate architecture, risks, and tradeoffs clearly to both technical and non-technical audiences.

Experience with cloud data platforms, APIs, streaming/event-driven architectures, AIP/AI-enabled workflows, or enterprise financial/operational data is beneficial.

This is a hands-on lead role. We are looking for someone who can both design the solution and work directly with the engineering team to build, review, stabilize, and continuously improve it.

Candidates must be authorized to work in the United States without current or future employer sponsorship.