[Remote] AI Performance Engineer

Note: The job is a remote job and is open to candidates in USA. Cornelis Networks is hiring an AI Performance Engineer to optimize training and multi-node inference across its networking silicon, systems, and software stack. The role measures and tunes distributed AI workloads, develops performance comparisons for customers, and shares best practices with internal and external audiences.

Responsibilities

• Measure & optimize end-to-end performance for distributed AI workloads (training + multi-node inference) across CPU & GPU clusters and diverse fabrics (Omni-Path, Ethernet, InfiniBand)
• Benchmark, characterize, and tune open-source & industry workloads and benchmarks (e.g., Llama, Qwen) on current and future compute, storage, and network hardware, including frameworks such as Megatron-LM, TorchTitan, Composer, vLLM, llama.cpp, etc
• Design and optimize distributed serving topologies (sharded/replicated, tensor/pipe parallel, MoE expert placement), continuous/adaptive batching, KV-cache sharding/offload (CPU/NVMe) & prefix caching
• Optimize training & inferencing: Validate RDMA/GPUDirect RDMA, congestion control, and collective/point-to-point tradeoffs during inference
• Build crisp proof points that compare Cornelis Omni-Path to competing interconnects; translate data into narratives for sales/marketing and existing customers
• Instrument and visualize performance (Nsight Systems, VTune, perf, app timers) plus database visualization (Prometheus/Grafana)
• Evangelize best practices through briefs, READMEs, blogs, articles, and peer-reviewed publications

Skills

• BS in CS/EE/CE/Math or related
• 2-3+ years running AI/ML at cluster scale
• Proven ability to set up, run, and analyze AI benchmarks; deep intuition for message passing, collectives, scaling efficiency, and bottleneck hunting for both training and inference
• Hands-on with distributed training beyond single-GPU (DP/TP/PP, ZeRO, FSDP, sharded optimizers) and distributed inference architectures (replicated vs sharded, tensor/KV parallel, MoE)
• Practical experience across AI stacks & comms: PyTorch, DeepSpeed, Megatron-LM, RCCL/NCCL, vLLM, llama.cpp
• Comfortable with compilers (GCC/LLVM/Intel/OneAPI) and MPI stacks; Python + shell power user
• Familiarity with network architectures (Omni-Path/OPA, InfiniBand, Ethernet/RDMA/RoCE) and Linux systems at the performance-tuning level, including NIC offloads, CQ moderation, pacing, ECN
• Excellent written and verbal communication—turn measurements into customer persuasion
• MS
• Cluster scheduler experience (e.g. SLURM, PBS)
• Hands-on profiling & tracing of GPU/comm paths (Nsight Systems, Nsight Compute, ROCm tools/rocprof/roctracer, VTune, perf, etc.)
• Background in HPC performance engineering or storage (BeeGFS, Lustre, NVMe-oF)

Benefits

• Fully remote work for employees residing within the United States.

Company Overview

Company H1B Sponsorship