Video by PyTorch via YouTube
Muon has attracted a lot of attention for fast convergence, but getting those optimizers to work in a real training stack is a different challenge.
At PyTorch Conference North America, Jen Wei will walk through Muon, Dion, and Dion3 and look at what happens when you add the messy details: sharding, communication, schedulers, and everything else that can go wrong.
If you are interested in optimizers and training dynamics, see you at PyTorchCon NA. Get your ticket: https://hubs.la/Q04v4SL60