Jongryool Kim - Disaggregated LLM Serving with Shared Memory KV Cache at Rack Scale
Berkeley RDI · 4:30 · Watch on YouTube
SK Hynix's Jung Wook Kim argues that a disaggregated, physically separate memory pool shared by several servers can speed up LLM serving. In SK Hynix's demo, the pool holds the KV cache so that prefill and decode node...