ML Platform Engineer, Distributed Training

Egen is seeking a skilled ML Platform Engineer to join a specialized team supporting a healthcare and life-sciences AI client. In this role, you will focus on the training stack, partnering with infrastructure teams to optimize, benchmark, and debug large-scale foundation model training on GKE-based GPU clusters.

Key responsibilities

• Collaborate with infrastructure teams to manage and optimize the GKE training platform, including GPU node pools and job scheduling.
• Run, scale, and debug multi-node distributed training workloads using PyTorch and Ray Train.
• Design and execute performance benchmarks to measure throughput, scaling efficiency, and cost-effectiveness.
• Build and maintain model evaluation harnesses to ensure platform changes are data-driven.
• Instrument training pipelines with telemetry and monitoring to ensure end-to-end reliability.

Requirements

• 5+ years of engineering experience with a focus on GPU training infrastructure and large-scale model training.
• Strong proficiency in Python, including writing production-quality code, testing, and code reviews.
• Hands-on experience with GKE, GPU scheduling, and capacity management on Google Cloud.
• Deep knowledge of distributed PyTorch, Ray Train, and networking protocols like RDMA, GPUDirect, and NCCL.
• Proven ability to own technical areas, proactively identify bottlenecks, and communicate complex design decisions.

What we offer

• The opportunity to work on cutting-edge foundation models within the healthcare and life-sciences sector.
• Full ownership of your technical domain with the ability to influence platform architecture and strategy.
• A collaborative environment working alongside expert client scientists and Egen engineers.
• Engagement in high-impact projects involving advanced hardware like H200 and GB200 clusters.