Guide

A collection of explanations for understanding distributed training.

1. Neural networks and training

   Parameters, loss, gradients, autodiff and the training loop.

2. Distributed training

   Data and model parallelism, what makes it hard, and Distributed Data Parallel.

3. DiLoCo and its variants

   Local steps with an outer optimizer, fragments, and rounds that do not wait for every worker.

4. GradientMail

   One round through the mailbox, connecting a training loop, and choosing the tradeoff.