Principal AI Platform Engineer

Job#: 3058816
Job Description:
About the Role
We are seeking a Principal AI Platform Engineer to help build, operate, and scale an enterprise AI platform supporting a growing portfolio of AI applications, agents, and business use cases.
This is a highly hands-on Principal-level position with significant technical ownership and organizational influence. You will own critical components of the AI platform, including AI gateways, GPU/model serving, RAG, observability, cost management, security, governance, and production operations.
This is an opportunity to join at an important stage of the organization's AI journey. You will help establish the technical standards, infrastructure, and operating model that future AI solutions are built against.
What You'll Be Doing

• Own key components of the enterprise AI platform from infrastructure through production operations.
• Build and operate an AI gateway supporting multiple models/providers, including routing, fallback, security, quotas, and cost allocation.
• Deploy and operate open-weight models on GPU infrastructure using technologies such as vLLM, NVIDIA Triton/TensorRT-LLM, or similar.
• Build and operate production RAG systems, including vector stores, embedding pipelines, document processing, retrieval, and governed knowledge sources.
• Establish AI observability and cost management, including visibility into token usage, GPU utilization, application/team consumption, performance, and overall platform cost.
• Implement tracing, monitoring, evaluation, and regression testing using tools such as Langfuse, LangSmith, Arize, or similar platforms.
• Deploy, monitor, troubleshoot, and support production AI agents and workflows.
• Evaluate model performance and economics to determine the appropriate balance between managed cloud models and self-hosted/open-weight solutions.
• Partner with engineering, infrastructure, security, operations, and business teams to operationalize AI safely and efficiently.
• Help establish standards around AI security, model access, RBAC, guardrails, governance, and production readiness.
• Provide technical guidance around model selection, routing, caching, batching, token optimization, and cost-versus-capability decisions.
• Communicate technical recommendations effectively with engineering teams, architects, and senior leadership.
• Remain hands-on with engineering and coding while providing Principal-level technical leadership.

Required Qualifications

• 6+ years of experience in Platform Engineering, Site Reliability Engineering, Infrastructure Engineering, or related technical operations.
• 2+ years running LLM/AI systems in production for real users, or significant ML platform operations experience.
• Hands-on experience implementing and operating an AI/LLM gateway in front of one or more model providers.
  • Examples include LiteLLM, Azure APIM, Kong, Portkey, or similar.
• Hands-on experience deploying open-weight models on GPUs.
  • Experience with vLLM, NVIDIA Triton/TensorRT-LLM, or comparable technologies strongly preferred.
• Experience building and operating RAG solutions in production.
• Experience with:
  • Vector stores
  • Embedding pipelines
  • Document processing
  • Azure AI Search, pgvector, or similar technologies
• Experience measuring and managing AI/LLM costs by application, workload, or team.
• Experience with AI observability, tracing, monitoring, and evaluation.
• Strong hands-on engineering skills with:
  • Python
  • Terraform and/or Bicep
  • CI/CD
  • Kubernetes
  • AKS or comparable Kubernetes environments
• Working knowledge of AI security risks, including prompt injection, data leakage, model abuse, access controls, and secure AI deployment practices.

Preferred Qualifications

• Azure cloud experience
• On-premise GPU infrastructure experience
• GPU sizing and optimization
• Quantization
• Fine-tuning / LoRA
• MCP
• LangChain / LangGraph
• Semantic Kernel
• AI agent frameworks and orchestration
• Experience building and shipping AI applications
• Experience using AI to improve or automate operational/support workflows
• Experience working within a regulated enterprise environment
• Experience establishing AI platform standards, governance, or enterprise frameworks

What We're Looking For
The ideal candidate has personally built and operated enterprise AI/LLM infrastructure in production and understands what happens after an AI solution moves beyond development and into ongoing operations.
We're particularly interested in candidates who can speak specifically about:

• Models they have deployed
• GPU infrastructure they have personally worked with
• Production scale and request volume
• RAG architectures they have built
• AI gateway architecture
• Actual monthly AI/cloud spend
• Cost optimization they personally implemented
• Production incidents and how they resolved them
• Users and workloads supported

This position requires real production ownership. Candidates should be able to provide concrete examples, metrics, and lessons learned from systems they personally built or operated.
Who May Not Be the Right Fit
This likely will not be the right opportunity for candidates who are primarily:

• Data Scientists without platform/infrastructure ownership
• Application Developers without significant infrastructure experience
• Engineers whose AI experience is primarily consuming model APIs
• Architects who are no longer hands-on
• Candidates whose AI experience is primarily personal projects or home labs
• Candidates without experience supporting AI/ML systems in a true production environment

Why This Opportunity Stands Out
This is not a maintenance role within an already mature AI organization. The person selected will have an opportunity to influence the foundation and direction of an evolving enterprise AI platform.
You'll combine hands-on engineering with platform ownership and have significant influence across:

• AI infrastructure
• Model strategy
• Platform architecture
• Cost optimization
• Observability
• Security
• Governance
• Production reliability
• Operational AI adoption

As the organization's AI capabilities continue to expand, there is also potential for the platform function and supporting team to grow over time.
Location & Work Environment
This is a remote position for candidates residing in an approved hiring state:
WA, OR, ID, AZ, FL, IL, MN, NC, OH, SC, or TX
Pacific Northwest candidates are preferred.
Occasional onsite collaboration may be requested based on business needs, with advance notice provided whenever possible.
Compensation & Benefits
The anticipated pay range for this position is $100.00 to $130.00 per hour. Please note that this pay range is a good faith estimate that may be impacted by several factors, including the candidate's qualifications, experience, and location. The pay range is subject to change at any time based on business needs.
This employer is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or protected veteran status and will not be discriminated against on the basis of disability.
Apex uses a virtual recruiter as part of the application process. Click for more details.
If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation in using our website for a search or application, please contact our Benefits Department at or .
Everforth Apex is a world-class IT services company that serves thousands of clients across the globe. When you join Everforth Apex, you become part of a team that values innovation, collaboration, and continuous learning. We offer quality career resources, training, certifications, development opportunities, and a comprehensive benefits package. Our commitment to excellence is reflected in many awards, including ClearlyRateds Best of Staffing in Talent Satisfaction in the United States and Great Place to Work in the United Kingdom and Mexico.
Everforth Apex uses a virtual recruiter as part of the application process. Click for more details. By applying for this job, you agree to receive calls, AI-generated calls, text messages, or emails from Everforth Apex and its affiliates, and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our privacy policy at
Everforth Apex Benefits Overview: Everforth Apex offers a range of supplemental benefits, including medical, dental, vision, life, disability, and other insurance plans that offer an optional layer of financial protection. We offer an ESPP (employee stock purchase program) and a 401K program which allows you to contribute typically within 30 days of starting, with a company match after 12 months of tenure. Everforth Apex also offers a HSA (Health Savings Account on the HDHP plan), a SupportLinc Employee Assistance Program (EAP) with up to 8 free counseling sessions, a corporate discount savings program and other discounts. In terms of professional development, Everforth Apex hosts an on-demand training program, provides access to certification prep and a library of technical and leadership courses/books/seminars once you have 6+ months of tenure, and certification discounts and other perks to associations that include CompTIA and IIBA. Everforth Apex has a dedicated customer service team for our Consultants that can address questions around benefits and other resources, as well as a certified Career Coach. You can access a full list of our benefits, programs, support teams and resources within our 'Welcome Packet' as well, which an Everforth Apex team member can provide.
Everforth Apex Systems is an equal opportunity employer. We do not discriminate or allow discrimination on the basis of race, color, religion, creed, sex (including pregnancy, childbirth, breastfeeding, or related medical conditions), age, sexual orientation, gender identity, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, disability, status as a crime victim, protected veteran status, political affiliation, union membership, or any other characteristic protected by law. Everforth Apex will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable law.
If you require an accommodation under the Americans with Disabilities Act to participate in an interview with a virtual recruiter or to use our website for a search or application, please contact our Benefits Department at or . Please note that this contact information is strictly to be used for medical ADA accommodations and that no other inquiries will be answered.
UnitedHealthcare creates and publishes the Transparency in Coverage Machine-Readable Files on behalf of Everforth Apex Systems.