Principal Core Infrastructure Engineer - GPU Validation

hackajob is collaborating with Oracle to connect them with exceptional professionals for this role.

Description

Join the HPC team to develop automated tools and validation frameworks that ensure GPU racks are ready for production. The role includes building NCCL performance benchmarks, GPU infrastructure and NVLink connectivity tests, Slurm-based validation workflows, and automated FDT and cable diagnostics.

Designs, implements, and optimizes components in distributed systems with an emphasis on scalability, resiliency, and operability.

Designs, implements, and optimizes components in distributed systems with an emphasis on scalability, resiliency, and operability. Delivers features and load/performance tests; leverages data plane platforms and distributed state tools for high-volume retrieval, storage, and processing; and reviews peers' implementations for scalability compliance. Builds fault-tolerant paths (redundancy, replication, automatic failover), applies recovery oriented principles, and implements retries, circuit breakers, and timeouts. Proactively detects and mitigates issues via tests, alarms, dashboards, and telemetry; authors runbooks and participates in incident response and RCAs. Implements standard replication and synchronization, develops automation/IaC for troubleshooting and maintenance, and applies advanced security controls (encryption, access, remediation) while ensuring change, compliance, and documentation standards are met.

Must live or be willing to relocate so you are able to regularly go into the Nashville, TN office.

Responsibilities

Key Responsibilities:

System Design & Validation Development

• Design and develop scalable validation frameworks for GPU racks and HPC infrastructure.
• Build automated benchmarks and diagnostics for NCCL performance, GPU health, NVLink connectivity, and InfiniBand and high-speed networks.
• Develop Slurm-based workflows for distributed validation and performance testing.
• Implement retries, timeouts, and recovery mechanisms to improve framework resilience.
• Develop functional, load, and fault-injection tests to assess reliability and production readiness.

Reliability & Operational Engineering

• Build dashboards, telemetry, and alerts to monitor infrastructure health and validation results.
• Develop automated tools for Linux diagnostics, log analysis, hardware fault isolation, FDT, and cable validation.
• Convert incident findings into reusable diagnostics and preventive automation.
• Develop recovery runbooks and contribute to incident response and root cause investigations.

Automation, Security & Change Management

• Develop and maintain Bash, Python, and Infrastructure as Code tools for deployment and configuration.
• Implement access controls and secure handling of infrastructure credentials and diagnostic data.
• Automate patching, updates, and rollback workflows in accordance with change management practices.
• Maintain technical documentation and improve automation coverage across infrastructure workflows.

Core Responsibilities:

Planning & Delivery

• Translate project requirements into development plans and deliverables.
• Manage assigned work with minimal supervision, meeting agreed timelines and quality standards.
• Adapt priorities as requirements, resources, and dependencies change.

Collaboration & Problem Solving

• Work with HPC, infrastructure, and engineering teams to define requirements and align on outcomes.
• Use logs, metrics, and diagnostic results to investigate issues and develop lasting solutions.
• Resolve issues independently, escalate when needed, and document findings to support shared understanding.

Continuous Learning & Improvement

• Build expertise in GPU systems, HPC technologies, and software development through training and feedback.
• Recommend and implement improvements to validation coverage, reliability, and development efficiency.
• Share reusable tools and best practices, and incorporate team feedback into ongoing improvements.

Must live or be willing to relocate so you are able to regularly go into the Nashville, TN office.

Qualifications

Disclaimer:

Certain U.S. based or U.S. customer or client-facing roles may be required to comply with applicable requirements, such as immunization/occupational health mandates, and/or drug testing requirements.
Range and benefit information provided in this posting are specific to the stated locations only

US: Hiring Range in USD from: $114,600 to $234,600 per annum. May be eligible for bonus, equity, and compensation deferral.

Oracle maintains broad salary ranges for its roles in order to account for variations in knowledge, skills, experience, market conditions and locations, as well as reflect Oracle's differing products, industries and lines of business.

Candidates are typically placed into the range based on the preceding factors as well as internal peer equity.

Oracle US offers a comprehensive benefits package which includes the following:

1. Medical, dental, and vision insurance, including expert medical opinion

2. Short term disability and long term disability

3. Life insurance and AD&D

4. Supplemental life insurance (Employee/Spouse/Child)

5. Health care and dependent care Flexible Spending Accounts

6. Pre-tax commuter and parking benefits

7. 401(k) Savings and Investment Plan with company match

8. Paid time off: Flexible Vacation is provided to all eligible employees assigned to a salaried (non-overtime eligible) position. Accrued Vacation is provided to all other employees eligible for vacation benefits. For employees working at least 35 hours per week, the vacation accrual rate is 13 days annually for the first three years of employment and 18 days annually for subsequent years of employment. Vacation accrual is prorated for employees working between 20 and 34 hours per week. Employees working fewer than 20 hours per week are not eligible for vacation.

9. 11 paid holidays

10. Paid sick leave: 72 hours of paid sick leave upon date of hire. Refreshes each calendar year. Unused balance will carry over each year up to a maximum cap of 112 hours.

11. Paid parental leave

12. Adoption assistance

13. Employee Stock Purchase Plan

14. Financial planning and group legal

15. Voluntary benefits including auto, homeowner and pet insurance

The role will generally accept applications for at least three calendar days from the posting date or as long as the job remains posted.

As part of Oracle's onboarding process and consistent with applicable law, US-based employees are required to complete identity verification, which involves the collection and processing of their biometric information. Accommodations to this requirement may be granted following an individualized assessment.