Haystack
← Back to Jobs
Technology
VC

Lead AI Engineer with Retirement OR Wealth Domain

Vish Consulting Services, Inc.United States🇺🇸United StatesPosted 1 Sept 2026

Quick Overview

Seniority
Mid Senior
Work mode
Hybrid
Location
United States
Posted
20 hours ago
DockerMongoDBNode.jsRustSQLSQL ServerAWSMLOpsMLflowSOC 2SnowflakeActixAirflowAzureData PipelineDatadogGPTGenerative AIGoogle CloudKubernetesLLMPostgreSQLPyTorchReactRedisTensorFlowTypeScriptdbt

Job Description

Responsibilities:-

·       Architecture & Technical Design

·       Hands-On Engineering

·       MLOps & Production Reliability

·       Technical Leadership

 

Experience

•            10+ years of progressive software engineering experience with sustained hands-on contributions (aligned with Citi C14/SVP benchmark for this level).

•            3+ years of dedicated experience building LLM-based systems and agentic architectures in production environments — not research or notebook work.

•            Proven success architecting and delivering multiple enterprise-scale AI solutions into production; can speak to architecture decisions, failure modes encountered, and how systems were improved post-launch.

•            Prior lead or staff-level role: set technical direction, owned critical systems end-to-end, influenced engineering practices across a team.

•            Experience delivering AI systems in a regulated environment (financial services, healthcare, or similar) with compliance, audit trail, and governance requirements.

 

Programming & Core Engineering

•            Rust (required, expert level): production systems development including memory safety, async programming with Tokio, error handling patterns, trait design, and testing — used for performance-critical AI service layers, data pipelines, and backend infrastructure.

•            TypeScript / Node.js (required): production API services, async/await patterns, type-safe API contracts, and React-based front-end interfaces for advisor and participant-facing tools; full-stack TypeScript capability is expected, not optional.

•            Solana / Solana programs (required): smart contract development using Anchor or native Solana program model; familiarity with Solana’s account model, transaction structure, and program-derived addresses (PDAs) as they apply to on-chain financial data and tokenized retirement or investment products.

•            Software engineering fundamentals: system design, CI/CD pipeline ownership, testing strategy (unit, integration, contract, eval), resiliency patterns, security practices for AI services, and operational stability.

•            API development: RESTful and event-driven API design using TypeScript/Node.js or Rust (Axum, Actix, or equivalent); authentication, rate limiting, versioning, and API contracts for AI services consumed by downstream systems.

•            Data engineering: complex SQL proficiency; data pipeline construction in Rust or TypeScript (dbt, Airflow, Prefect, or equivalent); working with structured financial data at scale; experience with Snowflake, Spark, or similar.

•            Front-end capability: React with TypeScript to build production-quality interfaces for advisor and participant-facing AI tools — not a specialization, but full ownership of the UI layer is expected.

•            Databases: vector databases (Pinecone, Weaviate, pgvector, OpenSearch); relational (PostgreSQL, SQL Server); document (MongoDB); caching (Redis).

 

LLM & Generative AI Engineering  —  Required

•            Production LLM integration: hands-on experience with OpenAI GPT-4o, Anthropic Claude, Google Gemini/Gemma, and/or AWS Bedrock in user-facing production applications — not just API experimentation.

•            RAG system design and implementation: vector store selection and configuration, chunking and embedding strategies, hybrid search, re-ranking, and rigorous evaluation (RAGAS, custom eval frameworks, or equivalent).

•            Prompt engineering at an engineering level: system prompt design for financial services safety constraints, few-shot construction, structured output extraction (JSON/XML), prompt version control, and regression testing.

•            Agentic AI architecture: tool use and function calling; multi-step reasoning chains; agent orchestration frameworks (LangGraph, LangChain, Google ADK, AutoGen, CrewAI, or custom implementations); MCP (Model Context Protocol) server design and integration for financial data sources.

•            LLM evaluation: building eval suites for correctness, hallucination, instruction-following, and task-specific quality; LLM-as-judge patterns; adversarial robustness testing for financial advice contexts.

•            Output validation and safety layers: guardrails, output parsers, confidence scoring, fallback logic, and human-in-the-loop escalation patterns for production AI systems handling regulated financial outputs.

•            ML frameworks: working knowledge of TensorFlow and PyTorch — sufficient to fine-tune, evaluate, and integrate transformer-based models; not required to build from scratch but must understand model mechanics to make architecture decisions.

 

Cloud, Infrastructure & MLOps

•            Cloud platforms: production experience on AWS, Azure, or Google Cloud Platform — AI/ML services (SageMaker, Azure ML, Vertex AI), serverless compute, managed databases, and storage.

•            Containerization and orchestration: Docker (required); Kubernetes working knowledge; experience deploying AI inference services in containerized environments with auto-scaling.

•            MLOps: experiment tracking (MLflow, Weights & Biases, or equivalent); model versioning; deployment pipelines for AI systems; CI/CD for model updates with automated quality gates.

•            Observability: logging, tracing, and metrics for AI services (Datadog, CloudWatch, OpenTelemetry, or equivalent); building dashboards and alerts for model quality, hallucination rates, and system health.

 

Retirement & Wealth Domain Knowledge  —  Required

Engineering in a retirement context is different from general fintech. You are building systems that touch fiduciary decisions and regulated investment outputs. You do not need to be a certified financial planner, but you must understand what you are building for.

 

•            Defined Contribution Mechanics: 401(k)/403(b)/457 plan structures; contribution and matching rules; the recordkeeper/TPA/plan sponsor data ecosystem; what participant data looks like, where it comes from, and what PII controls apply.

•            ERISA & Fiduciary AI Constraints: how fiduciary obligations constrain what AI systems can output autonomously vs. require human review; prohibited transaction implications for AI-generated investment guidance; audit logging requirements for ERISA-covered plan decisions.

•            2026 Model Risk Management Framework: the April 2026 interagency guidance replacing SR 11-7 — including its materiality-tiered approach, proportional controls, conceptual soundness documentation requirements, and principles-based treatment of GenAI and agentic systems; how to design AI systems that generate governance evidence as a byproduct of normal engineering operations.

•            Investment Data: fund data structures (NAV, returns, expense ratios, asset allocation); participant account and transaction data; market data feeds; familiarity with Aladdin (BlackRock), Orion, or similar platforms is a plus.

•            Privacy & Security in Financial AI: SOC 2 Type II requirements; PII handling for participant financial data; data residency and retention requirements; experience building AI systems with appropriate access controls for sensitive financial information.

 

Similar jobs