Senior Databricks Solution Architect

We are seeking a Senior Databricks Solution Architect to lead the architecture and technical execution of a Palantir Foundry to Databricks migration. The role involves assessing existing Palantir applications, mapping capabilities to Databricks services, defining the target-state Lakehouse architecture, and leading the migration of data engineering, AI/GenAI, analytics, chatbots, and business workflows into scalable, secure, and high-performance solutions.

Key Responsibilities

·    Define the target-state Databricks architecture, including scalable Lakehouse design across Bronze, Silver, and Gold layers.

·       Assess existing Palantir Foundry pipelines, datasets, applications, workflows, and AI capabilities; develop feature-by-feature mappings to Databricks and identify areas requiring redesign.

·       Lead the migration of data ingestion, transformation, data quality, validation, large-scale data comparison, and reconciliation capabilities.

·       Design and optimize data pipelines using Databricks Lakeflow, Spark Declarative Pipelines, Apache Spark, and Delta Lake, supporting batch and near-real-time processing.

·       Architect AI/GenAI solutions for structured and unstructured document extraction, LLMs, RAG, embeddings, vector search, model serving, and migration of Palantir AIP capabilities.

·       Design target-state solutions for dashboards, drill-down reporting, analytical applications, chatbots, conversational analytics, and business workflows.

·       Establish enterprise data governance, security, access controls, lineage, and auditing using Unity Catalog and appropriate RBAC policies.

·       Optimize Databricks workloads, data pipelines, and compute resources for scalability, performance, reliability, and cost efficiency.

·       Lead data engineering and AI/ML teams, conduct architecture reviews, prepare HLD/LLD documents and migration roadmaps, and collaborate with business stakeholders, Palantir SMEs, cloud architects, and application teams.

·       

Requirements

·    15+ years of IT experience.

·       Extensive experience in Databricks solution architecture, Lakehouse design, Delta Lake, Apache Spark, and enterprise data engineering.

·       Strong expertise in Bronze/Silver/Gold architecture, data ingestion, transformation, data quality, CDC, and batch/near-real-time processing.

·       Proven experience with Palantir Foundry and/or Palantir AIP, preferably involving platform migration, capability mapping, and application or workflow redesign.

·       Strong knowledge of AI/GenAI solutions, including LLMs, RAG, document intelligence, information extraction, embeddings, vector search, and model serving.

·       Hands-on experience with Unity Catalog, enterprise data governance, security, access controls, and cloud platforms such as AWS, Azure, or Google Cloud Platform.

·       Experience architecting analytics, dashboards, chatbots, application integrations, and business workflows in enterprise environments.

·       Demonstrated technical leadership, architecture documentation, performance optimization, stakeholder management, and ability to lead complex migration initiatives.

·       Bachelors degree in Computer Science, Engineering or related.