I'm glad there are more attempts at solving model routing, as costs (at API rates) has really become an issue. Some feedback:
1. Reiterate the cache issue from other comments already here. there is a lot of optimisation in harnesses around caching and a proxy model blows that up
2. Coding agents are model aware - they already route code discovery to mini / flash models, planning to heavy models, workflow design to ultra, implementation to mid / high etc. They know when they're exploring, planning, implementing, reviewing etc. and which model class to select and when it fails.
With a proxy you're breaking this control loop and feedback. It doesn't know, for ex. that it just attempted with deepseek v4 and it failed, lets try Opus?
3. How are you going to RL improvements and prevent the router becoming stale? You only have access to your own internal prompts and ~thousands of samples.
This is RL'd on one orgs codebase. There are going to be a lot of prompts you haven't seen before and have no insight to on how to route correctly, and you have no insight into users HF to improve your own model. Orgs aren't going to share their traces with you, so you need other sources to train on and improve
There are also new model releases every week that you need to keep up with - whats the story going to be here
4. Publish evals by running terminalbench / deepswe bench. Show us the performance / cost / time chart vs the other agent and model sets. If you can show gains there, you have a very simple value prop to sell where you can charge for a % of the saved costs
1. Agree it's important, fwiw the proxy model doesn't blow this up though - only incurs a 1 time cost when switching models and we're aware of that when making routing decisions
2. The agents are model aware yes but they are not incentivized to optimize too heavily here (in particular they don't use OS models even when they would be better). I think that's where this router comes in and brings genuine improvement.
3. Two parts here: 1 is continuing to grow our golden dataset over time, 2 is using reward signals from production traffic (on a per-customer basis or, if allowed, across all users)
4. Yes we have these internally, great callout that we should publish! Will do + will link from the repo soon. (Fwiw I think these benchmarks are useful but don't fully capture vibes - you should try it out yourself for that!)
The thing I do not get with these routers is that you will have more cache misses (5min ttl). And if there is one thing i’ve learned; using the cache is crucial.
How does this router translate to $$$ when developing?
You're right and that's why we built the router to be cache aware! Once it starts using one model, the threshold to switch to another model will be higher because the additional cost of the cache miss needs to be worth the cost savings or quality increase.
This is the key thing that other routers we've seen miss: they're stateless so for a coding agent use case you end up spending more money due to all the cache misses.
This is a key point. I don't know if you can still edit your submission, but I think this would be helpful to mention up front. I'm looking forward to testing this.
I'd say that a typical main agent loop has 1-3 models (obviously very situationally dependent), but when you have subagents those can get routed independently since they have a fresh context window, so there are a lot more degrees of freedom there.
The choice on the first turn is super important for this reason! But if a user prompt sends the convo in a very different direction then often it does make sense to reroute at that point.
Artefact-based workflows solve this problem, and I think it’s more effective to go in that direction.
I still have Claude Code because Opus makes good plans, but I hand the plan over to M3 on Pi with 99.9% cache hits on a long session. Lovely. Pi then makes a summary file that Opus can use to review the code/context.
But you do need them to write down their stuff, so that compaction and clear sessions can work off a nice, concise document.
And if you are simply using Claude Code, then /advisor is what you want: a sub-agent with a much cleaner context is spawned to handle something -> not cached per se, but much cheaper to run.
I’d stay away from workflows that automatically route between models unless you can afford the cache misses. That’s also why GLM 5.x is costing me much more, I don’t get good caching with it.
It's rather hard to do at the proxy level with agentic coding, such as Claude Code or similar. These are long-chained sessions of tool use that heavily rely on prompt caching. Changing mid-flight is costly.
It looks like much more context is required to decide on the best model (e.g., summarizing logs might use a cheap model, whereas you likely want Opus/Mythos/GPT 5.6 to debug multithreading logic). In an agentic system, a decision about the model may be embedded in the decision to orchestrate the model.
But intuitively I think it makes sense that a model can learn what model to route things to if it has all the relevant info, and experimentally it works pretty well in our experience
Man, I'm not so sure if I'd use something like this because the way I prompt already changes based upon what model I am using. I'm not convinced it would route to the right model based on my diction or whatever.
Yeah that's a really interesting point, tbh I think the more relevant variable here is the harness you're using rather than the specific model? i.e. GPT 5.5 in the Claude harness behaves a lot more like Claude than Codex if that makes sense.
Hard to quantify this ofc but that's what I've felt vibes wise from using this for the last month.
I have the same general feeling as well. Like you, I can’t prove it’s not just personal feeling - but e.x. Opus via Copilot CLI behaves entirely different than Opus via Claude Code, which behaves differently than Opus via OpenCode or Pi.
I have the same feeling. I've been trying Claude Code directly ever since Copilot nerfed their request-based system, and Opus just seems to perform "better" in Claude Code.
It's also possible that it's the 1m context versus the 200k context (Copilot's limit) doing some of the work here.
- vllm-sr/auto: efficient, fast, balanced routing, similar in spirit to Fugu // Sakana Fugu — Multi-Agent System as a Model: https://sakana.ai/fugu/
- vllm-sr/fusion: panel-style multi-model reasoning and synthesis.
- vllm-sr/flow: router-native workflow orchestration
- vllm-sr/remom: multi-round reasoning over one or multiple models.
Good questions. From what I can tell, vLLM semantic router is more optimized for one-off prompt/response workflows rather than agentic coding (I don't think it's cache aware).
As another commenter (https://news.ycombinator.com/item?id=48689994) pointed out, for one-off requests, I think it makes more sense to lock to one model whose behavior you understand very well. For dynamic requests like the ones going to a coding agent I think dynamic routing makes more sense but it does need to be cache aware.
I tried Sakana Fugu, boy is it hungry ... it blows up tokens like nothing I have ever seen. Not that impressed with the results I got from it however if I'm being honest. Now I'm bought into their buy 1 get 2nd month free so will keep trying it but may cancel after.
This would not work in the way that shows any significant genuine benefit IMO. Caching and optimum routing of a single request are at odds with each other. Higher the distinct model count in a conversation, more cache misses you accept.
Based on what OP said elsewhere in the discussion "threshold to switch to another model will be higher" means that essentially you reduce the workflow into two models at most. The two model primitive, one planner and one executor, is already sufficient for such a use case.
For lower than 2 models, it's just a simple single model cache-preserving conversation which arguably doesn't need another layer. For larger than 2 models, you are likely paying a large aggregate cache penalty that negates most of the gains
When we started building this we did it as an experiment and we thought the same thing might be true (cache misses would make the whole thing pointless). This turned out not to be true! I think there are 3 reasons intuitively:
1. Small models can carry out a good number of requests e2e
2. Small model for part of a request + cache miss < big model for entire request in many cases
3. Subagents
For our own usage we've saved 40% so far (that is of course including costs of uncached requests when switching models)
This assumes a perfect problem routing though. Determining the complexity class of an arbitrary problem is generally undecidable or extremely hard (Rice's theorem implication). So, in real use cases, you need to amortize all cases where the problem got routed to the wrong model and recovery had to be performed)
For example, if my task was "refactor this component to decouple all messy nesting", the problem router can't possibly know what is being referred to. This works for clear cut and dry problems but not for ambiguous problems. Most of the real world problems carry a lot of ambiguity.
I think the key detail here is that we use embeddings of the prompt + previous context in order to decide where to route the request, and if one model is getting stuck we can course correct and move to a different model.
So: we can reasonably cluster similar problems together and learn how models handle them, and the entire system doesn't fail if the initial decision is off.
In my mind one of the problems is that I'm using the term 'router' to describe something more akin to a train schedule. A list of abilities, cost, and timeframe to be used by a model capable of deconstructing its own process.
I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session.
I'm just trying to figure out why on the fly routing would beat testing and tuning and locking models and versions for each class of call, with evals and auto tunes running to explore more possible models for commonly run classes of prompt over time . . .
"Based on your subscription tier and local hardware here's a list of models that fit and process definitions your biggest brain will comfortably handle."
I guess that sounds a lot like moving your evals and auto tunes to a third-party, but I don't have the time, budget, or inclination to create a system like this out of whole cloth and then keep it relevant.
I could see something that provides on-the-fly routing information being useful, but actual decision-making is too dependent on context.
I see a great tension in the market today. On one hand you want agents to work reliably and that needs a lot of harness, computer use, model routine, tasks running longer etc. And on other hand you simply want to reduce your dependencies and costs.
Agent building is very nascent and all the frontier companies are trying to build the best harnesses possible.
As basic prompting, researching, coding gets mature, more and more of such tasks will be optimal for model routing open source etc etc but there is a chance that by that time frontier models again make costs and routing, low and effortless.
Basically I believe everyone has started jumping to the. -- REAL PROBLEM IS COST v. REAL PROBLEM IS EFFICIENT, RELIABLE AGENTS/WORKFLOWS.
It's going to be very interesting to see how this plays out.
I ran into a problem at work recently: we are given access to a bunch of models up to a full Claude Opus 4.8, but a monthly budget of 100k tokens. We are also given access to Gemini 3.5 Flash & 3.1 Pro with a daily budget of 50M tokens, but no tool calling. I'd love to hook Claude Code (or Pi) into the Gemini model, but the lack of tool-calling makes it quite difficult. I've been planning out how an intelligent router might be able to use a token-efficient tool-calling model (including a small local open-weights model) to handle the basic tools like reading from the file system or interfacing with MCP servers such that context is gathered, but then send the built up context to the Gemini model where I have a nearly unlimited (for my use cases) token budget.
I’m curious how a workplace ends up with a model policy like this. It seems like you’d spend more time trying to work out how to use a tiny number of Opus tokens than doing it yourself.
There are so many of these projects to wrangle AIs I think we might need an AI to go through, analysing each and amalgamating the good bits.
It makes me think of MakeFiles.
Make is sufficiently bad that everyone who has used it has considered writing a better way to do it. A good percentage of those people have done so.
On the other hand, make is also not so sufficiently bad that it cannot do its job. The choice becomes picking the thing that everyone has or one of the many many alternatives that proclaim their strengths and leave their weaknesses lurking to bite when they are least expected.
No single replacement to make dominates, and make lives on. I wonder if AI management is on a similar path.
Just curious how the router decides on which model to use. When I use Claude Code, I often ask Claude Code to decide itself if it should spawn a sub-agent to downgrade or upgrade the model. Claude Code is smart to know how much context and cache it has and will decide if it should use sub-agent with a lesser model (sometimes it costs more to re-fetch tokens with a Sonnet sub-agent if the parent agent already has the context).
I notice Cursor already does something similar. Even if I have Opus 4.8 selected, it will trigger subagents using Composer 2.5. I like using Auto personally because it is pretty effective and deeply discounted, but at work I YOLO Opus high.
I imagine a solution like this will eventually be an enterprise-forced solution because there is no reason right now for individual developers to be selective about model pricing. Even more important is non-tech users who do stuff through MCPs like "give me a full overview of all analytics" and let it chug for half an hour.
Oh interesting, didn't know Cursor did that! Totally makes sense though, routing subagents is def the easiest win, no need to have any cache awareness.
This would have been neat back when I could afford enough tokens to even set it up properly. Now I’ve had to increase my GH Copilot subscription just to cover the bare minimum updates to a few websites every month, and I no longer do any test driving, or even recreational coding projects. I don’t have hundreds of dollars a month to plow into these products, so I’m rationing use, looking for better local options, and being much more discerning about where these tools actually save time. Precarious time to be alive…
I've been building a reasonably complicated project over the past week using deepseek v4 pro almost exclusively (a couple of k2.7 and 1 session with gpt5.5 to re-assess some architectural questions). Deepseek is super capable though if you're a coder. I don't even write "code" but I can tell when it's doing something dumb and tell it how to do it better, but other than that I'm not micro managing it or using it "just for auto complete" or whatever.
Yes the open source models are very good, that’s a big part of what makes this router save so much money in practice! There definitely are some things they still don’t handle well though where you do want a frontier model
100%, from what we've seen, for a lot of big companies that 1. don't have subsidized usage and 2. are pushing AI adoption hard, figuring out token costs is P0 or P1 for their eng leadership
So you're saying that since adopting AI/LLM tech many companies have their top engineering priority being optimizing the costs of that rather than ... addressing actual business needs?
> with no noticeable differences in quality or velocity.
Have you done any A/B tests on this with evidence? (That's one thing I'd be very interested to see for claims like this - I'm not necessarily doubting you, it just seems like it could be useful to understand claims of quality/efficiency)
Great question! Our main product quantifies engineering productivity & quality so I think we're uniquely qualified to answer this - our velocity has only gone up and our quality (bugs introduced, code turnover) has not budged per our own analysis.
That is super curious - using more low quality cheaper models increased your velocity? My prior would have been slightly reduced velocity but massive reduction in token costs made it worthwhile.
I got Opus to knock out an MCP server that implements subagents running in pi and tell Opus to send work to DeepSeek. Or I tell it to ask GPT-5.5 for critiques. It's manual but saves a lot of tokens.
Is this talking to claude code, or to claude api (and paying api rates)? programatically routing requests through claude code sounds like a good way to get banned, just like the opencode and openclaw users.
Is that because you start by running it inside Claude Code? I don't see how Claude would allow any other harness to call them for their subscription, after all that OpenClaw hullabaloo.
Is this noticeably different than having your implementation planning phase break a larger task into sub-tasks, and recording the ideal model to use based on scope as part of the task definition?
Yes because it's a model explicitly trained to make model selections! Opus probably doesn't have a great idea of when to send a task to DeepSeek vs. to Sonnet, for example.
Large model companies will likely build this and make it better. It'll also be cheaper overall since they'll be subsidizing token cost if you use them directly vs third party router paying API costs
I would argue they do not have a good incentive to build this and make it better. Why would Anthropic route Claude Code traffic to DeepSeek (at 20% of the cost)?
So, how are you handling read/write caching? I mean, if I keep routing the next prompt based on the task weights? How about if I'm sending every 5th query to opus, which do expensive write cache?
We consider the cost of missing the cache when making each routing decision after the initial one. Discussed in a bit more depth here: https://news.ycombinator.com/item?id=48689448
What about request caching? If you swap to a cheaper model mid execution it might cost more that to make multiple requests to the already cached provider?
If you statelessly route each new request: yes it does end up being more expensive!
So our routing is cache-aware. It will have a much higher threshold to switch from one model to another if there's already some cache for the first model. Experimentally this solves the problem (like I said we've saved 40% ourselves vs. what we would have otherwise paid).
Indeed it is :) I skipped over talking about all the RL machinery, network design, reward function design, state representations, etc. because really the intuition is that we tell the model when it accomplishes its goal, and then it learns over time how to get better at making the right decisions in order to accomplish its goal.
Happy to talk about this in some more depth if there's anything specific you're curious about!
We haven't experimented with routing to local LLMs much. Technically they benefit from the cache too although it's more a question of latency than cost. But tbh I haven't seen great results in the wild from working with local LLMs for coding - curious if you've had any success with them?
I generally used them for token saving purposes, just using them for repetitive tasks, gated and supervised by claude. So its planned and verified by better models, but implementation falls on local ones. It has been pretty effective for me, as long as I spend a bit more initially on splitting complex tasks further down
We integrate with OpenCode too! OpenCode provides the harness, then the router selects the right model for the task.
We haven't yet set up local model routing though, that's really interesting - have you had any success using local models for coding tasks? Tbh I haven't heard many success stories from using local models yet
Cool.. but I still don't get how this is going to save money. It seems to me that it might actually burn more money just because the whole system now seems to be coming from different LLMs.
Also, small LLMs are prone to stop before completion, throw errors and produce loops. Is this factored in the design of the tool? I am not sure.
It saves money because some agent sessions can be entirely handled by a smaller model (also relevant: subagents use fresh context windows so a subagent with a simple task can be routed to a smaller model even if the main agent needs a frontier model).
Totally right about small LLMs btw, that's why we trained this on real agent sessions where we forced it to use different models. If the routing model sees small models can't handle a certain type of task then they won't be assigned. (Also as a fallback we have some guardrails that will have a bigger model come in to "rescue" a smaller model if it gets stuck)
A lot of people are working on the same thing because nobody's come up with a definition of "thing" that people agree on yet. Your project would be valuable just for adding another point of view to the conversaion.
Fun fact: Cursor's "auto" mode is just Composer (or at least it was last time I checked). So it's different in the sense that it actually does route to more than 1 model
If you have a Claude/Codex subscription then we use that (and account for the subsidized price accordingly when making routing decisions) instead of API billing. So you get the best of both worlds: subsidized usage for frontier models + save by using open/smaller models when it's genuinely better.
In practice, lots of ppl are using this to make their Claude sub limits go further!
But we're not routing via `claude -p`, if you have sub usage available + it's the right choice to route to a Claude model, then the router is approximately a transparent passthrough. So it gets billed like normal `claude` usage rather than `claude -p`.
Oh cool, feel free to reach out to me at andrew@workweave.ai if you ever want to share notes! We've learned a lot in the process of building this so far :)
We have created Murmur[1] which kind of works with your existing subscription (having API key is not mandatory). You can just tag @copilot @codex from claude code to delegate work to them. (it can also do it on its own too btw)
Curious what's the typical switching frequency in your experiments and experience. How do you control the tradeoff of cache matching and model efficiency?
It's a real concern! We take this stuff super seriously (https://trust.mycroft.io/weave) and tbh most of our customers opt for the hosted version because it's much simpler on their end + they're already trusting us with a bunch of sensitive data.
But of course since the source is available you can also run it locally or self host
I'm glad there are more attempts at solving model routing, as costs (at API rates) has really become an issue. Some feedback:
1. Reiterate the cache issue from other comments already here. there is a lot of optimisation in harnesses around caching and a proxy model blows that up
2. Coding agents are model aware - they already route code discovery to mini / flash models, planning to heavy models, workflow design to ultra, implementation to mid / high etc. They know when they're exploring, planning, implementing, reviewing etc. and which model class to select and when it fails.
With a proxy you're breaking this control loop and feedback. It doesn't know, for ex. that it just attempted with deepseek v4 and it failed, lets try Opus?
3. How are you going to RL improvements and prevent the router becoming stale? You only have access to your own internal prompts and ~thousands of samples.
This is RL'd on one orgs codebase. There are going to be a lot of prompts you haven't seen before and have no insight to on how to route correctly, and you have no insight into users HF to improve your own model. Orgs aren't going to share their traces with you, so you need other sources to train on and improve
There are also new model releases every week that you need to keep up with - whats the story going to be here
4. Publish evals by running terminalbench / deepswe bench. Show us the performance / cost / time chart vs the other agent and model sets. If you can show gains there, you have a very simple value prop to sell where you can charge for a % of the saved costs
Really appreciate the thoughtful feedback!
1. Agree it's important, fwiw the proxy model doesn't blow this up though - only incurs a 1 time cost when switching models and we're aware of that when making routing decisions
2. The agents are model aware yes but they are not incentivized to optimize too heavily here (in particular they don't use OS models even when they would be better). I think that's where this router comes in and brings genuine improvement.
3. Two parts here: 1 is continuing to grow our golden dataset over time, 2 is using reward signals from production traffic (on a per-customer basis or, if allowed, across all users)
4. Yes we have these internally, great callout that we should publish! Will do + will link from the repo soon. (Fwiw I think these benchmarks are useful but don't fully capture vibes - you should try it out yourself for that!)
The thing I do not get with these routers is that you will have more cache misses (5min ttl). And if there is one thing i’ve learned; using the cache is crucial.
How does this router translate to $$$ when developing?
You're right and that's why we built the router to be cache aware! Once it starts using one model, the threshold to switch to another model will be higher because the additional cost of the cache miss needs to be worth the cost savings or quality increase.
This is the key thing that other routers we've seen miss: they're stateless so for a coding agent use case you end up spending more money due to all the cache misses.
This is a key point. I don't know if you can still edit your submission, but I think this would be helpful to mention up front. I'm looking forward to testing this.
That is interesting, sounds like in practice you only end up routing between 2 models
I'd say that a typical main agent loop has 1-3 models (obviously very situationally dependent), but when you have subagents those can get routed independently since they have a fresh context window, so there are a lot more degrees of freedom there.
Or not routing at all.
In practice you just pick one and stick with it until the API stops or you hit performance issues.
The choice on the first turn is super important for this reason! But if a user prompt sends the convo in a very different direction then often it does make sense to reroute at that point.
Artefact-based workflows solve this problem, and I think it’s more effective to go in that direction.
I still have Claude Code because Opus makes good plans, but I hand the plan over to M3 on Pi with 99.9% cache hits on a long session. Lovely. Pi then makes a summary file that Opus can use to review the code/context.
But you do need them to write down their stuff, so that compaction and clear sessions can work off a nice, concise document.
And if you are simply using Claude Code, then /advisor is what you want: a sub-agent with a much cleaner context is spawned to handle something -> not cached per se, but much cheaper to run.
I’d stay away from workflows that automatically route between models unless you can afford the cache misses. That’s also why GLM 5.x is costing me much more, I don’t get good caching with it.
It's rather hard to do at the proxy level with agentic coding, such as Claude Code or similar. These are long-chained sessions of tool use that heavily rely on prompt caching. Changing mid-flight is costly.
It looks like much more context is required to decide on the best model (e.g., summarizing logs might use a cheap model, whereas you likely want Opus/Mythos/GPT 5.6 to debug multithreading logic). In an agentic system, a decision about the model may be embedded in the decision to orchestrate the model.
Yep cache awareness is super important, mentioned this in another thread here: (https://news.ycombinator.com/item?id=48689448)
But intuitively I think it makes sense that a model can learn what model to route things to if it has all the relevant info, and experimentally it works pretty well in our experience
Man, I'm not so sure if I'd use something like this because the way I prompt already changes based upon what model I am using. I'm not convinced it would route to the right model based on my diction or whatever.
Yeah that's a really interesting point, tbh I think the more relevant variable here is the harness you're using rather than the specific model? i.e. GPT 5.5 in the Claude harness behaves a lot more like Claude than Codex if that makes sense.
Hard to quantify this ofc but that's what I've felt vibes wise from using this for the last month.
I have the same general feeling as well. Like you, I can’t prove it’s not just personal feeling - but e.x. Opus via Copilot CLI behaves entirely different than Opus via Claude Code, which behaves differently than Opus via OpenCode or Pi.
I have the same feeling. I've been trying Claude Code directly ever since Copilot nerfed their request-based system, and Opus just seems to perform "better" in Claude Code.
It's also possible that it's the 1m context versus the 200k context (Copilot's limit) doing some of the work here.
> Man, I'm not so sure if I'd use something like this because the way I prompt already changes based upon what model I am using.
Perhaps you're just not the best use case. It may work better when Average Joe is the one prompting.
Yep this was always the reason to avoid "auto" mode in cursor.
Looks interesting!
Out of curiosity, how does it compare with vLLM Semantic Router?
For reference:
https://vllm-semantic-router.com/
https://github.com/vllm-project/semantic-router
vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models, https://arxiv.org/abs/2603.04444
https://github.com/vllm-project/semantic-router
For instance, does it offer similar algorithms:
- vllm-sr/auto: efficient, fast, balanced routing, similar in spirit to Fugu // Sakana Fugu — Multi-Agent System as a Model: https://sakana.ai/fugu/ - vllm-sr/fusion: panel-style multi-model reasoning and synthesis. - vllm-sr/flow: router-native workflow orchestration - vllm-sr/remom: multi-round reasoning over one or multiple models.
FWIW, it does look good on https://routeworks.github.io/leaderboard
Ref.
RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers, https://arxiv.org/abs/2510.00202, https://github.com/RouteWorks/RouterArena
Good questions. From what I can tell, vLLM semantic router is more optimized for one-off prompt/response workflows rather than agentic coding (I don't think it's cache aware).
As another commenter (https://news.ycombinator.com/item?id=48689994) pointed out, for one-off requests, I think it makes more sense to lock to one model whose behavior you understand very well. For dynamic requests like the ones going to a coding agent I think dynamic routing makes more sense but it does need to be cache aware.
I tried Sakana Fugu, boy is it hungry ... it blows up tokens like nothing I have ever seen. Not that impressed with the results I got from it however if I'm being honest. Now I'm bought into their buy 1 get 2nd month free so will keep trying it but may cancel after.
This would not work in the way that shows any significant genuine benefit IMO. Caching and optimum routing of a single request are at odds with each other. Higher the distinct model count in a conversation, more cache misses you accept.
Based on what OP said elsewhere in the discussion "threshold to switch to another model will be higher" means that essentially you reduce the workflow into two models at most. The two model primitive, one planner and one executor, is already sufficient for such a use case.
For lower than 2 models, it's just a simple single model cache-preserving conversation which arguably doesn't need another layer. For larger than 2 models, you are likely paying a large aggregate cache penalty that negates most of the gains
When we started building this we did it as an experiment and we thought the same thing might be true (cache misses would make the whole thing pointless). This turned out not to be true! I think there are 3 reasons intuitively:
1. Small models can carry out a good number of requests e2e 2. Small model for part of a request + cache miss < big model for entire request in many cases 3. Subagents
For our own usage we've saved 40% so far (that is of course including costs of uncached requests when switching models)
This assumes a perfect problem routing though. Determining the complexity class of an arbitrary problem is generally undecidable or extremely hard (Rice's theorem implication). So, in real use cases, you need to amortize all cases where the problem got routed to the wrong model and recovery had to be performed)
For example, if my task was "refactor this component to decouple all messy nesting", the problem router can't possibly know what is being referred to. This works for clear cut and dry problems but not for ambiguous problems. Most of the real world problems carry a lot of ambiguity.
I think the key detail here is that we use embeddings of the prompt + previous context in order to decide where to route the request, and if one model is getting stuck we can course correct and move to a different model.
So: we can reasonably cluster similar problems together and learn how models handle them, and the entire system doesn't fail if the initial decision is off.
In my mind one of the problems is that I'm using the term 'router' to describe something more akin to a train schedule. A list of abilities, cost, and timeframe to be used by a model capable of deconstructing its own process.
I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session.
I'm just trying to figure out why on the fly routing would beat testing and tuning and locking models and versions for each class of call, with evals and auto tunes running to explore more possible models for commonly run classes of prompt over time . . .
"Based on your subscription tier and local hardware here's a list of models that fit and process definitions your biggest brain will comfortably handle."
I guess that sounds a lot like moving your evals and auto tunes to a third-party, but I don't have the time, budget, or inclination to create a system like this out of whole cloth and then keep it relevant.
I could see something that provides on-the-fly routing information being useful, but actual decision-making is too dependent on context.
I see a great tension in the market today. On one hand you want agents to work reliably and that needs a lot of harness, computer use, model routine, tasks running longer etc. And on other hand you simply want to reduce your dependencies and costs. Agent building is very nascent and all the frontier companies are trying to build the best harnesses possible. As basic prompting, researching, coding gets mature, more and more of such tasks will be optimal for model routing open source etc etc but there is a chance that by that time frontier models again make costs and routing, low and effortless. Basically I believe everyone has started jumping to the. -- REAL PROBLEM IS COST v. REAL PROBLEM IS EFFICIENT, RELIABLE AGENTS/WORKFLOWS. It's going to be very interesting to see how this plays out.
I ran into a problem at work recently: we are given access to a bunch of models up to a full Claude Opus 4.8, but a monthly budget of 100k tokens. We are also given access to Gemini 3.5 Flash & 3.1 Pro with a daily budget of 50M tokens, but no tool calling. I'd love to hook Claude Code (or Pi) into the Gemini model, but the lack of tool-calling makes it quite difficult. I've been planning out how an intelligent router might be able to use a token-efficient tool-calling model (including a small local open-weights model) to handle the basic tools like reading from the file system or interfacing with MCP servers such that context is gathered, but then send the built up context to the Gemini model where I have a nearly unlimited (for my use cases) token budget.
Could your router handle this?
I’m curious how a workplace ends up with a model policy like this. It seems like you’d spend more time trying to work out how to use a tiny number of Opus tokens than doing it yourself.
Let's just say this organization is very, very large and doesn't necessarily have the budget for everyone to have all of the tokens.
Yes we can route to Gemini models too and we handle all the translation complexity there!
Monthly budget of 100k Opus tokens? So $2.50 worth?
There are so many of these projects to wrangle AIs I think we might need an AI to go through, analysing each and amalgamating the good bits.
It makes me think of MakeFiles.
Make is sufficiently bad that everyone who has used it has considered writing a better way to do it. A good percentage of those people have done so.
On the other hand, make is also not so sufficiently bad that it cannot do its job. The choice becomes picking the thing that everyone has or one of the many many alternatives that proclaim their strengths and leave their weaknesses lurking to bite when they are least expected.
No single replacement to make dominates, and make lives on. I wonder if AI management is on a similar path.
Just curious how the router decides on which model to use. When I use Claude Code, I often ask Claude Code to decide itself if it should spawn a sub-agent to downgrade or upgrade the model. Claude Code is smart to know how much context and cache it has and will decide if it should use sub-agent with a lesser model (sometimes it costs more to re-fetch tokens with a Sonnet sub-agent if the parent agent already has the context).
We trained a model to select which LLM to call at any given turn, based on lots of agent traces
I notice Cursor already does something similar. Even if I have Opus 4.8 selected, it will trigger subagents using Composer 2.5. I like using Auto personally because it is pretty effective and deeply discounted, but at work I YOLO Opus high.
I imagine a solution like this will eventually be an enterprise-forced solution because there is no reason right now for individual developers to be selective about model pricing. Even more important is non-tech users who do stuff through MCPs like "give me a full overview of all analytics" and let it chug for half an hour.
Oh interesting, didn't know Cursor did that! Totally makes sense though, routing subagents is def the easiest win, no need to have any cache awareness.
This would have been neat back when I could afford enough tokens to even set it up properly. Now I’ve had to increase my GH Copilot subscription just to cover the bare minimum updates to a few websites every month, and I no longer do any test driving, or even recreational coding projects. I don’t have hundreds of dollars a month to plow into these products, so I’m rationing use, looking for better local options, and being much more discerning about where these tools actually save time. Precarious time to be alive…
> Now I’ve had to increase my GH Copilot subscription
Maybe you should move away from a subscription that started charging by the token instead of by the request?
I've been building a reasonably complicated project over the past week using deepseek v4 pro almost exclusively (a couple of k2.7 and 1 session with gpt5.5 to re-assess some architectural questions). Deepseek is super capable though if you're a coder. I don't even write "code" but I can tell when it's doing something dumb and tell it how to do it better, but other than that I'm not micro managing it or using it "just for auto complete" or whatever.
And it is SO fucking cheap.
Yes the open source models are very good, that’s a big part of what makes this router save so much money in practice! There definitely are some things they still don’t handle well though where you do want a frontier model
try OpenCode
This + making sure common requests are saved as reusable skills and scripts would probably save a large part of my token usage
As prices increase we will see more of these tools to optimise and make the best use of token budget
100%, from what we've seen, for a lot of big companies that 1. don't have subsidized usage and 2. are pushing AI adoption hard, figuring out token costs is P0 or P1 for their eng leadership
So you're saying that since adopting AI/LLM tech many companies have their top engineering priority being optimizing the costs of that rather than ... addressing actual business needs?
I guess delivering business value is always #1, I just meant it's the biggest problem they're trying to solve. Here's a recent example that was public: https://fortune.com/2026/05/26/uber-coo-ai-spending-tokens-c...
Do you reward the RL model based on the token consumption when multiple LLMs complete the task ?
Effectively yes (based on cost though, not raw token count)
> with no noticeable differences in quality or velocity.
Have you done any A/B tests on this with evidence? (That's one thing I'd be very interested to see for claims like this - I'm not necessarily doubting you, it just seems like it could be useful to understand claims of quality/efficiency)
Great question! Our main product quantifies engineering productivity & quality so I think we're uniquely qualified to answer this - our velocity has only gone up and our quality (bugs introduced, code turnover) has not budged per our own analysis.
> our velocity has only gone up
That is super curious - using more low quality cheaper models increased your velocity? My prior would have been slightly reduced velocity but massive reduction in token costs made it worthwhile.
Is that due to the faster inference time?
I got Opus to knock out an MCP server that implements subagents running in pi and tell Opus to send work to DeepSeek. Or I tell it to ask GPT-5.5 for critiques. It's manual but saves a lot of tokens.
Is this talking to claude code, or to claude api (and paying api rates)? programatically routing requests through claude code sounds like a good way to get banned, just like the opencode and openclaw users.
If you have a Claude sub with subsidized usage we use that. If not you pay API prices.
Is that because you start by running it inside Claude Code? I don't see how Claude would allow any other harness to call them for their subscription, after all that OpenClaw hullabaloo.
Yep exactly
Is this noticeably different than having your implementation planning phase break a larger task into sub-tasks, and recording the ideal model to use based on scope as part of the task definition?
Yes because it's a model explicitly trained to make model selections! Opus probably doesn't have a great idea of when to send a task to DeepSeek vs. to Sonnet, for example.
Large model companies will likely build this and make it better. It'll also be cheaper overall since they'll be subsidizing token cost if you use them directly vs third party router paying API costs
I would argue they do not have a good incentive to build this and make it better. Why would Anthropic route Claude Code traffic to DeepSeek (at 20% of the cost)?
They'll route traffic to Haiku or one of their cheaper models, not third parties. Overall cost will end up being cheaper than whatever you are doing
We welcome the competition :)
So, how are you handling read/write caching? I mean, if I keep routing the next prompt based on the task weights? How about if I'm sending every 5th query to opus, which do expensive write cache?
We consider the cost of missing the cache when making each routing decision after the initial one. Discussed in a bit more depth here: https://news.ycombinator.com/item?id=48689448
Looks cool, but I couldnt find any benchmarks for how well this does. Any links?
I generally believe the proxy route is best to understand any harness. I been building some thing similar.
What about request caching? If you swap to a cheaper model mid execution it might cost more that to make multiple requests to the already cached provider?
Yep 100%, mentioned this in another thread (https://news.ycombinator.com/item?id=48689448) but tl;dr we build the router to be cache aware
Isn't this more expensive than always using the same model, since, as I understand, by routing to different models you give up on cache?
If you statelessly route each new request: yes it does end up being more expensive!
So our routing is cache-aware. It will have a much higher threshold to switch from one model to another if there's already some cache for the first model. Experimentally this solves the problem (like I said we've saved 40% ourselves vs. what we would have otherwise paid).
"We reward the routing model when it selects an LLM that achieves the task successfully" sounds pretty oversimplified
Indeed it is :) I skipped over talking about all the RL machinery, network design, reward function design, state representations, etc. because really the intuition is that we tell the model when it accomplishes its goal, and then it learns over time how to get better at making the right decisions in order to accomplish its goal.
Happy to talk about this in some more depth if there's anything specific you're curious about!
> At Weave, we write ~all our code with AI
This is probably not a very effective way of marketing imo. At least, it turns me completely off.
Fair enough, not meant to be marketing just a statement of fact. Would have turned me off too 18 months ago but times change...
Can't really win can ya? Scarce? They're driving up prices! Plentiful? It's all a big bubble!
This is cool!
Will this use my Claude Pro/Max subscription? Or will it always use the API billing "pay as you go"?
Yep it uses the Claude sub if possible and falls back to API billing only if you don't have a Claude sub or it's out of usage! Same deal for Codex
This might be a stupid question, but can a extra added local llm help with the caching problem?
We haven't experimented with routing to local LLMs much. Technically they benefit from the cache too although it's more a question of latency than cost. But tbh I haven't seen great results in the wild from working with local LLMs for coding - curious if you've had any success with them?
I generally used them for token saving purposes, just using them for repetitive tasks, gated and supervised by claude. So its planned and verified by better models, but implementation falls on local ones. It has been pretty effective for me, as long as I spend a bit more initially on splitting complex tasks further down
Wont this kill the kv cache?
Also i am pretty sure neither open ai or anthropic leets you seed the agents own tokens.
Very important consideration, addressed it in another thread (https://news.ycombinator.com/item?id=48689448). tl;dr we built this to be cache aware for exactly this reason
I would rather just use OpenCode - leverage AI models, even can host locally or paid ones with ease.
We integrate with OpenCode too! OpenCode provides the harness, then the router selects the right model for the task.
We haven't yet set up local model routing though, that's really interesting - have you had any success using local models for coding tasks? Tbh I haven't heard many success stories from using local models yet
Cool.. but I still don't get how this is going to save money. It seems to me that it might actually burn more money just because the whole system now seems to be coming from different LLMs.
Also, small LLMs are prone to stop before completion, throw errors and produce loops. Is this factored in the design of the tool? I am not sure.
edit: spellcheck
It saves money because some agent sessions can be entirely handled by a smaller model (also relevant: subagents use fresh context windows so a subagent with a simple task can be routed to a smaller model even if the main agent needs a frontier model).
Totally right about small LLMs btw, that's why we trained this on real agent sessions where we forced it to use different models. If the routing model sees small models can't handle a certain type of task then they won't be assigned. (Also as a fallback we have some guardrails that will have a bigger model come in to "rescue" a smaller model if it gets stuck)
Ahh been working on the same thing for a while now but haven't launched yet
A lot of people are working on the same thing because nobody's come up with a definition of "thing" that people agree on yet. Your project would be valuable just for adding another point of view to the conversaion.
Cool, interested to see your approach when you do launch! I think it's a really interesting problem
this is impressive. genuinely better than most people appraoches with using LLM as another judge to help route. which just uses more tokens than saves
Appreciate the kind words! Lmk if you have any feedback on it from using!
What is the difference from Cursors 'auto' mode?
Fun fact: Cursor's "auto" mode is just Composer (or at least it was last time I checked). So it's different in the sense that it actually does route to more than 1 model
How did you check? Like looking at the results or at the actual implementation?
I mean, I know that it mostly chooses Composer, but I wonder if it is hard-wired or if they have a logic that just selects Composer most of the time?
but this means you work with API pricing rather than subscription pricing. Isn’t it better to use claude or codex CLI etc directly in terms of cost?
If you have a Claude/Codex subscription then we use that (and account for the subsidized price accordingly when making routing decisions) instead of API billing. So you get the best of both worlds: subsidized usage for frontier models + save by using open/smaller models when it's genuinely better.
In practice, lots of ppl are using this to make their Claude sub limits go further!
I see but didn’t they severely limited the usage allowed with `claude -p`
But we're not routing via `claude -p`, if you have sub usage available + it's the right choice to route to a Claude model, then the router is approximately a transparent passthrough. So it gets billed like normal `claude` usage rather than `claude -p`.
It is funny. We are building something similar.
Oh cool, feel free to reach out to me at andrew@workweave.ai if you ever want to share notes! We've learned a lot in the process of building this so far :)
We have created Murmur[1] which kind of works with your existing subscription (having API key is not mandatory). You can just tag @copilot @codex from claude code to delegate work to them. (it can also do it on its own too btw)
1. https://github.com/instavm/murmur - Murmur
Very interesting - curious how you've used it yourself so far? I can imagine one use case would be having e.g. GPT 5.5 review Opus 4.8's work?
Useful in splitting a big task - some parts are easy so give it to say Gemini. Some are harder so give it to gpt 5.5 and so on.
Also the throughput kind of increases since providers are different.
What I want is a router that can also provision compute on demand and shut it down when it is done.
Curious what's the typical switching frequency in your experiments and experience. How do you control the tradeoff of cache matching and model efficiency?
This is basically what I need, a router. I’m tired of changing intelligence & speed levels manually.
Nice, let me know any feedback you have from trying it out!
How come data privacy and confidentiality is not an issue with services like these?
Do people voluntarily use these proxies/routers, knowing their prompts, outputs and code will be seen by other people ?
I get it might be ok for personal projects, but for anything that makes money and is a part of business... this must be big no-no ?
It is a router that runs locally.
It's a real concern! We take this stuff super seriously (https://trust.mycroft.io/weave) and tbh most of our customers opt for the hosted version because it's much simpler on their end + they're already trusting us with a bunch of sensitive data.
But of course since the source is available you can also run it locally or self host