How Thousands of GPUs Train One AI: 5 Ways They Work Together
Jia-Bin Huang · 40:27 · Watch on YouTube
Training a frontier model such as Llama 3 405B needs 16,384 GPUs because weights, optimizer state, and activations cannot fit or stay busy on one parallelism axis. The workable design stacks tensor, sequence, context,...