Let's reproduce GPT-2 (124M)
Andrej Karpathy · 241:26 · Watch on YouTube
Andrej Karpathy implements GPT-2 (124M) from scratch in PyTorch, loads OpenAI’s weights to verify the architecture, then trains a randomly initialized copy on FineWeb-Edu with GPT-3-style optimization and shows it can...