AIAWS AI1h ago
Introducing Amazon SageMaker HyperPod Inference Gateway
Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing add-on for Amazon EKS. It uses real-time GPU signals to send each inference request to the best-suited pod, cutting first-token latency by up to 82% with no changes to your model servers or…
Read full articleSource: AWS AI · Opens in new tab