Principal Engineer

Role : Principal Engineer

Location : Charlotte NC (Onsite)

Persistent systems

Role Overview

As a Principal Engineer for AI Tools, you will sit at the critical intersection of applied artificial intelligence, developer experience (DevEx), and high-scale distributed systems. You will define the technical strategy and reference architecture for internal AI tools—including autonomous agent hosting, AI-driven code review systems, internal vector/knowledge retrieval bases, and LLM orchestration gateways. This is a senior individual contributor (IC) leadership role focused on multiplying engineering efficiency, establishing security and evaluation best practices, and shaping how hundreds or thousands of engineers build software using AI-native workflows.

Key Responsibilities

Platform Architecture & Tooling Strategy

• Design internal AI platforms: Architect scalable, low-latency internal developer tools, IDE extensions, cloud agent hosting infrastructures, and automated code review pipelines.
• Set technical standards: Define reusable architectural patterns, governance-by-design frameworks, and SDK abstractions for secure LLM consumption across product and engineering teams.
• Drive build vs. buy decisions: Evaluate emerging foundational models, inference engines, vector databases, and developer productivity ecosystems to optimize cost, latency, and quality tradeoffs.

LLM & Intelligent Systems Integration

• Orchestrate AI workflows: Integrate large language models (LLMs), retrieval-augmented generation (RAG), and agentic planning loops into developer and operational tooling.
• Optimize retrieval & context: Build high-performance knowledge retrieval platforms that allow AI tools to safely query internal codebases, documentation, and system telemetry.
• Implement guardrails & evaluation: Deploy rigorous evaluation frameworks to measure tool accuracy, catch behavioral drift, minimize hallucinations, and guard against data leakage or prompt injection.

Technical Leadership & Mentorship

• Multiply team capability: Mentor senior and staff engineers, raise the standard of engineering craftsmanship, and lead through technical influence rather than authority.
• Partner across organizations: Collaborate closely with product management, security, infrastructure, and developer productivity teams to align tool roadmaps with company-wide goals.
• Champion AI-native engineering: Evangelize best practices for AI-assisted development, prompt workflows, and generative AI operations (GenAIOps) across the company.

Required Qualifications & Competencies

Experience & Background

• 10+ years of professional software engineering experience, with at least 5 years operating in a senior, lead, or principal capacity building large-scale backend platforms or developer tooling.
• Demonstrated track record of architecting and shipping complex distributed systems or AI-integrated platforms into production environments.

Technical Hard Skills

• Proficiency in core languages: Expert-level programming skills in Python, Go, Rust, or C++.
• AI & LLM Ecosystem: Deep command of LLM orchestration frameworks (e.g., LangChain, LlamaIndex), vector databases (e.g., Pinecone, Milvus, Qdrant), and RAG architecture design.
• Cloud & Infrastructure: Strong command of containerization and orchestration tooling (Docker, Kubernetes) and modern cloud environments (AWS, Google Cloud Platform, or Azure).
• Systems Design: Exceptional distributed systems design skills, focusing on high availability, low latency, multi-tenancy, and high-throughput data processing.

Core Competencies

• Agentic AI workflows and tool invocation patterns
• Semantic search and hybrid retrieval systems
• Observability, monitoring, and tracing (e.g., OpenTelemetry, MLflow)
• AI security alignment, guardrails, and compliance

Preferred Qualifications

• Experience building internal developer platforms (IDPs) or developer productivity tools at scale.
• Hands-on exposure to fine-tuning smaller open-source models (Llama, Mistral) for specialized domain tasks.
• Contributions to open-source developer ecosystems or AI frameworks