techl8

Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine

Running large language model inference at scale forces a KV cache trade-off: oversized GPU instances or slow time-to-first-token. This post builds a tiered KV cache on Amazon SageMaker HyperPod that extends the cache into a shared, distributed NVMe pool with Curvine, so replicas…

Source: AWS AI · Opens in new tab