[Remote] ML Platform Engineer, Distributed Training

Note: The job is a remote job and is open to candidates in USA. Egen is hiring an ML Platform Engineer to work embedded with a healthcare and life-sciences AI client training foundation models on a large GKE-based GPU cluster. The role focuses on supporting and tuning the training platform, running and debugging distributed training, benchmarking and evaluating performance, and partnering with infrastructure teams and scientists.

Responsibilities

• Work with the client's infrastructure team on the GKE training platform (GPU node pools, job queueing and scheduling, quota and capacity) from the training workload's side
• Diagnose and fix platform failures at the root (for example a controller that keeps running out of memory) rather than restarting them
• Tune multi-node GPU networking (RDMA/GPUDirect, NCCL) for training throughput
• Run, scale and debug multi-node distributed training with PyTorch (FSDP, DeepSpeed, Megatron or NeMo-style) and Ray Train
• Profile jobs and remove bottlenecks across compute, memory, input pipelines and interconnect
• Design and run performance benchmarks for the cluster and training stack: throughput, scaling efficiency, cost per run
• Build and maintain model evaluation harnesses so changes to the platform or training recipes are measured, not assumed
• Instrument training and the platform with telemetry and monitoring, and keep end-to-end pipeline tests running against what users actually run
• Treat the platform as your own: proactively find issues, plug gaps and propose solutions before anyone asks, keep the team informed of status and risks, and look after the scientists who use it
• Explain and defend your design decisions to client engineers and scientists in detail
• Stay effective while the client's priorities and team structure change

Skills

• 5+ years of engineering experience, with recent hands-on work on GPU training infrastructure and large-scale training
• Strong Python software engineering: production-quality code, tests and code review
• GKE and GPU scheduling: Kubernetes on Google Cloud with GPU node pools, job queueing (Kueue or similar), and capacity and quota management
• Distributed PyTorch and Ray Train: has personally run and debugged multi-node training jobs
• RDMA and GPU networking: GPUDirect/RDMA, NCCL tuning, multi-node GB200 (or H200) topology
• Benchmarking and evals: has designed performance benchmarks and model evaluations, and used them to make decisions
• Deep enough understanding to explain how and why a system works, without leaning on AI coding tools to carry the reasoning
• A track record of visible ownership: present, communicative, and accountable on a demanding team
• Observability and telemetry for training platforms: metrics, tracing, alerting, end-to-end pipeline testing
• Healthcare or life-sciences data (genomics, clinical, biomedical), and working under HIPAA

Benefits

• Comprehensive Health Insurance
• Paid Leave (Vacation/PTO)
• Paid Holidays
• Sick Leave
• Parental Leave
• Bereavement Leave
• 401 (k) Employer Match
• Employee Referral Bonuses

Company Overview

Company H1B Sponsorship