Inference at Scale: Building the Infrastructure for AI’s Next Phase
Grace Gong · 18:31 · Watch on YouTube
Scalable inference is not just about raw throughput: for voice agents and real-time video models, latency, P99 behaviour, KV-cache memory movement, specialised hardware, energy and proximity to the user all become the...