Senior Operational Engineer – AI Infrastructure

Job Title: Senior Operational Engineer – AI Infrastructure
Experience: 5–10+ Years
Duration: Long-Term Contract
Location: Houston, TX; New York, NY; San Francisco, CA; Seattle, WA

Job Summary

We are seeking an experienced Senior Operational Engineer – AI Infrastructure to support the reliability, availability, and performance of production AI infrastructure services. The ideal candidate will have a strong background in infrastructure operations, incident management, monitoring and alerting, IT service management (ITSM), and operational process improvement.

The candidate will be responsible for monitoring production environments, coordinating incident response, managing service requests, maintaining operational documentation, and collaborating with GPU infrastructure, platform engineering, networking, hardware operations, and Site Reliability Engineering (SRE) teams.

This role also focuses on improving service reliability, streamlining operational workflows, and implementing automation to reduce manual effort and prevent recurring incidents.

Key Responsibilities

• Manage end-to-end incident and service-request lifecycles, including ticket creation, categorization, prioritization, assignment, escalation, status updates, documentation, and verified closure.

• Monitor production infrastructure health, respond to alerts, assess service impact, coordinate troubleshooting, and communicate incident status to relevant stakeholders.

• Maintain operational runbooks, standard operating procedures (SOPs), shift handoff documentation, escalation matrices, on-call schedules, and service ownership records.

• Collaborate with GPU infrastructure, platform engineering, networking, hardware operations, and SRE teams to troubleshoot issues and minimize service disruptions.

• Coordinate infrastructure maintenance, change readiness, incident reviews, root cause analysis (RCA), and post-incident corrective actions.

• Track operational metrics, incident trends, service-level agreements (SLAs), service-level objectives (SLOs), and reliability performance.

• Develop and maintain operational dashboards, reports, and workflow improvements to enhance visibility and efficiency.

• Identify repetitive manual processes and implement automation using Python, Bash, APIs, and workflow integration tools.

• Ensure operational records, incident timelines, and technical documentation remain accurate and aligned with established procedures.

• Drive continuous improvement initiatives focused on incident response, service quality, operational consistency, and production readiness.

Required Qualifications

• 5+ years of experience supporting production infrastructure, cloud services, platform operations, Site Reliability Engineering (SRE), Network Operations Center (NOC), or infrastructure engineering environments.

• Hands-on experience with incident management, service requests, ITSM ticketing workflows, escalation procedures, and issue resolution through verified closure.

• Experience with infrastructure monitoring, alert management, production health dashboards, on-call response, and incident coordination.

• Working knowledge of Linux systems, networking fundamentals, cloud infrastructure, and production troubleshooting.

• Experience with IT service management and collaboration tools such as Jira, Jira Service Management, ServiceNow, or equivalent platforms.

• Familiarity with monitoring and incident response tools such as Grafana, PagerDuty, metrics, alerts, and dashboards.

• Strong documentation skills, including maintaining runbooks, operational procedures, incident timelines, and shift handoff records.

• Excellent communication, prioritization, problem-solving, and cross-functional coordination skills, particularly during high-priority incidents.

• Ability to analyze operational data, identify recurring issues, and recommend improvements to service reliability and incident response.

Preferred Qualifications

• Experience supporting AI infrastructure, GPU compute clusters, high-performance computing (HPC), bare-metal servers, or data center IT operations.

• Familiarity with GPU infrastructure operations, GPU nodes, hardware incident workflows, and cross-team escalation processes.

• Scripting and automation experience using Python, Bash, REST APIs, or workflow integration tools.

• Experience supporting 24/7 production operations, on-call rotations, and multi-team incident bridges.

• Knowledge of SLA/SLO reporting, service reliability metrics, root cause analysis, and corrective action tracking.

• Experience improving ITSM workflows, operational dashboards, incident reporting, and service management processes.