Search

Machine Learning Engineer - Distributed Training & Research Infrastructure

PublishedPublished: 6/14/2022
Technology
|

Job Description

We’re partnering with a leading global trading firm to hire a Machine Learning Engineer to build the systems their researchers use to train models at scale. You’ll own the infrastructure that turns a research idea into a trained, tested model, and you’ll be measured on how fast and how cheaply that happens.

\n


\n

What you’ll do

\n

    \n
  • Build and scale distributed training pipelines for large models and complex, high-volume datasets
  • \n

  • Design the parallelism strategy, data loading and checkpointing for multi-node GPU training
  • \n

  • Automate the research loop with quantitative researchers: experiment orchestration, hyperparameter tuning and model retraining
  • \n

  • Work with HPC specialists to raise training throughput and GPU utilization, and to cut compute cost
  • \n

  • Extend the internals of PyTorch and other open-source ML tools when off-the-shelf isn’t fast enough
  • \n

  • Evaluate third-party tools and decide when to build versus buy
  • \n

\n


\n

What you’ll bring

\n

    \n
  • 3+ years in machine learning, focused on training systems
  • \n

  • Hands-on experience scaling training across many GPUs and nodes with PyTorch, JAX or TensorFlow (e.g. FSDP, DeepSpeed, Megatron, DDP)
  • \n

  • A strong grasp of ML fundamentals, including transformer and attention internals, beyond high-level APIs
  • \n

  • Production-quality Python, with testing discipline and ownership of the systems you ship
  • \n

  • A habit of measuring before optimizing: profiling, benchmarking and finding the real bottleneck
  • \n

  • Clear communication, so you can explain your technical decisions to researchers and adapt when the evidence says to
  • \n

\n


\n

Nice to have

\n

    \n
  • Experience with HPC schedulers (Slurm) and cloud or orchestration platforms (Kubernetes, Ray)
  • \n

  • Contributions to open-source ML or distributed-systems projects
  • \n

  • Familiarity with NCCL, collective communication or interconnect tuning
  • \n

\n


\n

No trading experience is required. What matters is depth in large-scale training systems and the ability to move fast among very smart people.

Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...