GradientMail

Low-communication distributed training without P2P connections

Decoupled DiLoCo over mailboxes (S3, R2, MinIO, HF Storage Buckets)

Read the guide → Introduction

Four learners, A to D, each put a delta in a shared mailbox. Learner B, the elected syncer, gets the deltas, merges them and commits the fragment, and every learner aligns with the commit.

Install

uv add "gradientmail[s3,torch]"
# or
pip install "gradientmail[s3,torch]"

Python 3.12+. Use mlx instead of torch for MLX on Apple Silicon.

Documentation

Introduction

How it works, installing, adding a session to a training loop, and the examples.
Guide

A collection of explanations for understanding distributed training.
Patterns

Ways to use GradientMail’s mailbox for more than parameter exchange.
Posts

A Pythia-14M run on four rented GPUs, MLX and PyTorch learners in one run, merging deltas with RDA, and implementing a mailbox backend.