Back to Jobs
Recently postedTechnology

AI Infrastructure Deployment Engineer

InOptra Digital Innovations Inc.Union City, CA🇺🇸United States

Quick overview

SeniorityMid Senior
Work modeOn Site
LocationUnion City, CA, United States
MLOpsAirflowLinux

Job description

AI Infrastructure Deployment Engineer
Location: United States. Must be US-based and willing to work on customer sites.
Role
Builds what the architect designed, and proves it works. Hands-on from rack and stack through to signed-off acceptance: physical installation, firmware and out-of-band configuration, fabric bring-up, cluster software install, and the validation and burn-in that demonstrates the cluster performs to specification.
This is a build-and-prove role. The validation half matters as much as the installation half, because an OEM-branded GPU build does not come with a ready-made acceptance suite — NVIDIA certification validates server models and configurations, not the cluster standing on a customer floor. The engineer runs and, where needed, assembles that acceptance testing.
Responsibilities
Physical build
    • Install GPU servers and GPUs; verify seating, airflow and serviceability.
    • Identify and validate cables and transceivers across the compute, storage and management fabrics; confirm signal quality before handover.
    • Confirm power and cooling parameters against the design, and escalate where the data hall will not support it.
Bring-up and configuration
    • Configure BMC and out-of-band management; set up TPM where required.
    • Apply and verify firmware across GPU baseboards, NICs, switches, DPUs and transceivers; interpret and act on fault detection output.
    • Install and configure the control plane: NVIDIA Base Command Manager including high-availability head nodes, operating system and cluster components, Slurm with Enroot and Pyxis, GPU and DOCA drivers, container toolkit and NCLI.
    • Set initial parameters for third-party storage and confirm throughput against the design.
Fabric
    • InfiniBand: provisioning, high availability, partition keys for multi-tenancy, quality of service, adaptive routing, and link and bandwidth monitoring through UFM.
    • Or Spectrum-X Ethernet: RoCE, ECN and PFC configuration, quality of service, adaptive routing, telemetry.
    • Troubleshoot with the standard tooling — ibdiagnet, iblinkinfo, ib_write_lat and ib_write_bw, ibping, What Just Happened, UFM health.
Validation and acceptance — the half that is usually missing
    • Single-node stress and burn-in.
    • HPL and single-node NCCL, including NVLink and NVLink Switch verification.
    • ClusterKit node assessment to find the weak node before the customer does.
    • NCCL east-west fabric bandwidth tests across the cluster.
    • Sustained burn-in using NCCL, HPL and NeMo workloads.
    • Storage throughput testing under realistic checkpoint load.
    • Produce the acceptance report: what was tested, what it returned, what was remediated.
Operate and remediate
    • Diagnose and replace failed GPUs, network cards, fans and power supplies.
    • Tune storage and fabric where results fall short of specification.
    • Hand over to the customer or operations team with documentation that matches what was actually built.
Experience and credentials
    • 5–8 years in data centre infrastructure engineering, of which at least 2–3 years hands-on with NVIDIA GPU hardware. This matches NVIDIA's own stated prerequisite for its AI infrastructure certification.
    • Delivered at least one multi-node GPU cluster build. Node count, GPU model, fabric type and the candidate's own scope must be stated, not implied.
    • Comfortable on a customer site, under programme pressure, working to someone else's design and method of procedure.
    • Linux systems fluency; scripting for repeatable build steps.
Certifications — strongly preferred:
    • NVIDIA-Certified Professional: AI Infrastructure (NCP-AII)
    • NVIDIA-Certified Professional: AI Networking (NCP-AIN) for fabric-leaning candidates
What takes a candidate out of scope
    • Remote-only or lab-only experience. This work happens on site.
    • Data scientists, ML engineers and MLOps specialists.
    • Pure network engineers with no server, GPU or firmware exposure, unless hired explicitly against the fabric specialism.
    • Candidates who cannot name the clusters they built and what they personally did on them.

Will your resume get past this employer's screening? Check your resume against this job, free

Similar jobs