Why LLMs speak in tokens

Name: Why LLMs speak in tokens
Uploaded: 2026-10-25T00:00:00.000Z
Duration: 7 min 29 s
Description: Large language models never see raw words: text is tokenized into subword pieces, those pieces become numeric IDs, and each ID looks up a learned vector before the neural net runs. The same path is used to prepare tra...

Ben Dicken · 7:29 · Watch on YouTube

Large language models never see raw words: text is tokenized into subword pieces, those pieces become numeric IDs, and each ID looks up a learned vector before the neural net runs. The same path is used to prepare tra...

Read the full summary on retube