Briefs · Taipei time (GMT+8)

SageMaker AI real-time endpoints add KV cache-aware routing

一句重點:SageMaker AI's new routing for real-time endpoints directs incoming prompts to the instance with the largest matching portion already stored in its KV cache.

發生什麼事

AWS Machine Learning Blog introduced KV cache-aware routing for SageMaker AI real-time endpoints. The feature selects an instance based on how much of a prompt it has cached. The article explains the routing behavior, compares when to use it with prefix-aware routing, and describes how to enable it.

01Sources

Evidence: 🗣 Official statement; not independently tested.