Nvidia GPU infrastructure Engineer/ HPC Engineer
NR Consulting - India · Pune, TN
Strong hiring record · median 12 days open
- First seen
- 13 November 2026
- Reposted
- Not since September 2026
- Last checked
- Today · still on the employer's board
All reasons
In its favour
- Posted 3 days ago
- Still listed on the employer's own board
Title
Nvidia GPU infrastructure Engineer/ HPC Engineer
Location
Pune
Exp
7-12 Yrs
Job Description
Key Responsibilities
Implementation
- Stand up multi-pod GPU clusters, including rack, power, cooling layouts, TOR/leaf connectivity, and InfiniBand/Ethernet host configurations.
- Implement GPU partitioning using MIG/vGPU profiles and quota policies for multi-tenant environments.
- Integrate cluster schedulers such as Slurm and Kubernetes with GPU device plugins, Node Feature Discovery, accounting, and quota management.
Operations
- Manage Day 2 operations across firmware, drivers, DCGM, and NVML lifecycles.
- Execute infrastructure change windows with zero or minimal downtime.
- Perform capacity planning for GPU, CPU, memory, and NIC resources.
- Implement bin-packing strategies for heterogeneous GPU SKUs.
Performance & Optimization
- Tune NCCL/UCX, GPU clocks and persistence, GPU Direct Storage, NUMA/locality, and CUDA runtime parameters.
- Drive benchmarking and acceptance using HPL, HPL-AI, and MLPerf-like internal benchmark suites.
- Monitor performance regressions and maintain performance SLOs.
Reliability & Incident Management
- Lead P0/P1 incident responses involving GPU, CUDA, and scheduler-related issues.
- Act as the primary technical lead during production outages, coordinating with Networking, GPU Operations, Platform Engineering, Storage, and Application teams to restore services within SLA targets.
- Perform detailed Root Cause Analysis (RCA) for network, GPU, CUDA, NCCL, RDMA, and scheduler-related failures.
- Identify underlying issues and implement preventive and corrective actions.
- Collaborate with platform and GPU engineering teams to resolve issues impacting CUDA jobs, Kubernetes scheduling, Slurm workload management, GPU resource allocation, and large-scale AI training environments.
- Define golden images and implement node remediation processes, including cordon, drain, reimage, and auto-healing workflows.
Security & Compliance
- Enforce GPU tenancy isolation using MIG, cgroups/device cgroups, and MPS daemon (mpsd).
- Ensure secure drivers and containers.
- Implement SBOM and image scanning practices.
Documentation & Enablement
- Publish operational runbooks and performance baselines.
- Develop application tuning guides for LLM training and inference workloads.
Experience & Educational Requirements
Qualifications and Experience
Educational Qualifications
- BE/B.Tech or equivalent degree in Computer Science or Electronics & Communication.
Relevant Experience
- Relevant experience in GPU cluster infrastructure, GPU operations, and large-scale AI/ML environments.
- Hands-on experience with NVIDIA GPUs, CUDA, NCCL, UCX, InfiniBand/RDMA, and GPU resource management.
- Experience with Kubernetes, Slurm, GPU scheduling, multi-tenancy, and performance optimization.
- Experience in production incident management, troubleshooting, and Root Cause Analysis (RCA).
Skills
- Slurm
- Kubernetes
- Node.js
- CUDA
- RDMA
- LLM
More jobs at NR Consulting - India
All 212- Salesforce QANR Consulting - India · Pan India2d ago
- Salesforce Senior Developer – AgentforceNR Consulting - India · Pan India2d ago
- JavaNR Consulting - India · Bangalore, pune, Chennai, Hyd2d ago
- GCP DevOps EngineerNR Consulting - India · Pune, MH2d ago
- IBM Sterling OMS DeveloperNR Consulting - India · Pan India2d ago
Similar roles
- Senior Systems Administrator IIJobgether · US · USD 91,074–137,670/yrtoday
- Sr Network EngineerJobgether · UStoday
- Senior Site Reliability EngineerJobgether · UStoday
- Corporate IT ManagerThe Lumber Manufactory · Philadelphia · USD 105,000/yrtoday
- Junior Full Stack DeveloperHelfy · Tbilisi, Georgiatoday
- IT Desktop Support TechnicianAa2it · Reedsville, PA · USD 84–170/hrtoday