Video by PyTorch Foundation via YouTube
Scaling PyTorch on Intel XPU: Field Notes From AI for Science on Aurora – Sam Foreman, Argonne National Laboratory; Panagiotis Kourdis, Intel; Tanima Dey, Intel Corporation
Bringing a new accelerator up as a first-class PyTorch backend at scale takes more than vendor patches. It requires reproducible benchmarking, portable training code, and operational machinery that survives long-running jobs. This lightning talk shares experiences from training scientific foundation models in production on 63K+ XPUs on ALCF’s Aurora system.
We cover three things:
– What was needed: the XPU integration path (torch.xpu, the IPEX-to-xccl transition, distributed primitives, collective tuning) and the cross-vendor abstractions that let identical training code run on Intel, NVIDIA, and AMD hardware.
– What was accomplished: migrating our production training stack from a Megatron-DeepSpeed fork to torchtitan + DTensor + FSDP2, with optimization and testing at every layer; from fused kernels and collectives up through distributed orchestration.
– What remains: fused kernels, async checkpointing, fault tolerance, and the operational layer long jobs require (silent-hang detection, bad-node failover, layered restart loops), plus what the next accelerator team should expect.
The takeaway: Patterns for anyone running PyTorch at scale on a new accelerator.
"