Sors: a Rust proxy that reorders prompts to maximize vLLM prefix cache hits

(github.com)

2 points | by flaccount 6 hours ago ago

No comments yet.