The Most Absurd Way To Train LLMs... With 3x Less Memory!?
bycloud · 16:08 · Watch on YouTube
Sakana AI’s Diffusion Blocks method, as explained here, treats a transformer’s residual depth as a denoising trajectory so each chunk of layers can be trained on its own noise range, cutting the slice of the model tha...