Hoping this is real. It's too bad to see the signals from Qwen that they will not be releasing a 35B-A3B for the 3.8 lineup. The MoE architecture makes a huge difference for being able to run these local models on reasonable consumer hardware.
Honest question/suggestion for the HN audience- Since Qwen released the weights for Qwen3.8 2.4T-A95B and we already have the staring point of Qwen3.6 35B-A3B, couldn't someone distill the bigger model and make a "pseudo" Qwen3.8 35B-A3B? Sure, it wouldn't be an official Qwen release but couldn't someone improve on Qwen 3.6 and get the thing everyone is asking for?
I am calling this a suggestion for the audience because I don't have the will/resources to do this.
> Base / architecture: Qwen/Qwen3.6-35B-A3B (Qwen3_5MoeForCausalLM, 256 experts, ~3B active). The "3.8" in the name refers to the teacher, not the base.
Not endorsement, haven't run it myself, just found the link.
The smarter 27B is so fast with MTP I've found I really don't need the 35B-A3B. You get around 70tk/s on a M5 Max lowering to around 40tk/s at higher context sizes.
I've benched 3.8 27B being significantly slower and less quality than 3.6 35B-A4B (both 4-bit quant, MTP, both using turboquant 4-bit served by oMLX), to the point that I'm not even using it right now (on an M3 Max). What's your use case and what did you observe? I might be missing something.
I believe you mean 35B-A3B, there was no such thing as A4B. I use 27B and other models for software development, and quite a few research or similar agents. I cannot imagine a world where the old 35B-A3B model is smarter / more capable than 3.8 27B - the difference is night and day for coding at least. Where 35B-A3B was fast and felt like a Haiku model, 27B feels like a strong Sonnet when given the right tools. I don't use turbo quant so can't comment on that, but with the A3B model you're using you probably won't get much from using MTP with small MoE models like that.
People over-quantize things, muck with the temperature and other settings based on superstitions or results from models they think are similar. There's lots of ways to make 3.8 27B dumber.
I've had great success with allowing the model to review optimal settings and my system specs. It comes up with the right configuration. Running Pi harness.
I just had Ornith 1.5 take a moment to configure itself, now it's reviewing a large project I'm working on, so far its really impressive for my needs. Qwen3.8 27b Unsloth(Dynamic 3.0) is also perfect. These two are working together, and I'm in a sweet-spot, I now have all I need.
Yes, A3B, sorry for the typo. I’ve benched it and used it in practice for agentic coding, and I haven’t seen any benefit to it yet. Lightning MTP helps a lot in the perf department.
I would suggest careful benchmarking. I actually tested and benchmarked, and the new Qwen3.8-27B model is actually slower with MTP on my M4 Max. MTP only gains anything when generating long code sequences, which is very unlikely as the model spends most of its time thinking, not generating code, even if you use it for coding (which I don't).
I get ~10 tok/sec on a M1 Ultra, but the real issue is abysmal prompt processing which makes it unusable. On the M1 and M2 series MTP is actively harmful for performance.
27B means it has 27 billion parameters, but how that translates to model size depends on the model architecture. For Qwen 3.5 architecture (same as 3.8) the native size is around 55 GB memory. So you need quantization to fit it in 32 GB.
See the right hand side panel, you see a whole palette of quantizations and their respective sizes. Should give you an idea. Note that these are not the only quantizations available.
Quick tests suggest it’s pretty good at reasoning and tool use (keen to search to check its thinking) and it seems to waste much less time thinking, too.
So it feels very fast.
But it does not seem to be better than Qwen 3.6 35B at coding. A bit worse, I think, though I will test it more.
If you have a machine that can fit a 35B model in VRAM, I would suggest testing Muse Glimmer with (from memory)
Reasoning strength: low
in the system prompt.
Despite being a dense model, this is actually capable of solving code problems faster than the Qwen MoE, despite having only one fifth of the raw token performance.
Personally I find speculative decoding much better strategy than MoE – performance wise it's there at 90-100 t/s on 2x4090, great intelligence – really great fit.
A lot of people, including me, don’t want to bother with GPUs, they’d rather run it on their M1-M5 MacBook. For example the 35B-A3B is very usable even on a M1 64GB MacBook.
Speculative decoding also works on Mac, 64G is more than what I have, m5 max should handle up to ~40 t/s with optimized setup (and with a lot of vram you can get great wins on concurrency – that harness can take advantage of for single user task as well), but agree memory bandwidth in mac or spark is still too slow, next gen for both will be great hardware to have for sure.
MoE is great on systems that lack the VRAM to host the full model. On my 16GB VRAM system, I can get 100 tok/s with Q4 Qwen 3.6 35b a3b, and 15 tok/s with 27b.
MTP is a trade-off, as it pushes some more of the model off the GPU.
I have managed to get usable quants of Laguna S2 and even DeepSeek V4 flash on this setup.
There is clearly some intelligence loss compared to similar sized dense models, but I feel like it stomps on the 9-12b models I could run fully on GPU
> The MoE architecture makes a huge difference for being able to run these local models on reasonable consumer hardware
That's not true. For computers without unified memory architecture (which is the vast majority) VRAM capacity is the bottleneck for local models. In that case a dense model can deliver significantly more intelligence than an MoE model of the same size. And for a typical consumer/gamer Nvidia GPU, dense models are fast enough.
Expert offloading significantly helps with the VRAM capacity.
Most MoE architectures have a few experts that are always running; this, the router, KV, and whatever else you have space for can stay in fast VRAM; and the remaining experts can be offloaded.
How is ornith-1.5's base model developed? Is the base model one of the Open weights models, or one pre trained by ornith team from scratch? I couldn't find information to answer this question in the article.
I've been using the 35B-A3B today for some web scraping work, and it has been on par with Qwen3.8 27B at a much higher speed and at a higher quant (q4 vs q8). I'm impressed.
What you say is in agreement with someone I follow on X. My family has been traveling for two weeks and I have been looking forward to trying qwen3.8-37b when I get home.
Now I might try Ornith-1.5-35B first even though I don’t see an official MLX version.
I had to go down to UD-Q3_K_XL for Qwen 3.8 27B to get it to fit in VRAM and be usable, but I worry I'm gutting its intelligence somewhat. I too am interested in faster + more-usable alternative that can exchange blows with the Q3-dumbed 27B.
They may have utility in trying to look at the whole landscape of models, but are very misleading when it comes to making 1:1 comparisons or in developing confidence at to how a given model will deliver on your workflow.
Only relevant benchmarks are those you make yourself, targeted specifically for your workflows. Anything else is just number go up on a pretty graph, and every model out there is probably benchmaxxed to hell on the public ones anyway. Keep yours private.
These models have gotten a fair amount of attention -- we're hoping it's enough to get them added to some reliable inference providers and OpenRouter, at which point we'll run them on our full benchmark suite.
Interestingly, in my own benchmark and testing (in the hopes of finding a good-enough local model to run a personal assistant agent), Ornith-1.0-9B was worse than Qwen3.5-9B which according to their scores should've been reversed.
I will definitely pass Ornith-1.5-9B through the gauntlet as well!
Can someone parse all that AI generated blather in the post and tell me clearly:
1. Is this self improvement at the model level (updated weights or memory, KV etc) or just by adding agentic code harnesess to guide the output better?
I was looking for something more general-purpose, such as a decommissioned quad-socket x86 with 80 or more AVX512 cores and a terabyte of RAM, which I suspect will be a strong limiting factor. The possibility of adding some older Nvidia accelerators is interesting as well even though they won’t be able to hold the whole model in HBM and will need to stream it over PCIe 4, which sucks.
That way, when not running inference, the machine can easily host VMs for other experiments and general housekeeping functions.
Nobody (with the probable exception of Anthropic given their work on character training) really trains models on their identity and Claude is the only AI persona that's well-defined so if you put yourself into the AI's shoes it's a pretty reasonable guess that it might be Claude. I've had basically every open model claim it's Claude when the topic comes up.
Anthropic changed their character training policy many times, and the name is probably separate from it anyway (besides the bits from the constitution etc). Name training usually comes last in all models, if at all, and it's pretty shallow. Certain Claude models say they are Qwen or Deepseek when asked in Chinese, for example.
Yeah, Claude is actually surprisingly unsure of his identity considering that their most recent publication on their constitutional AI training literally had graphs demonstrating how certain properties differed based on whether they were phrased as questions about "Claude" versus "You", but it's actually that which makes me fairly confident that they're probably doing _something_ to try and close that gap.
More generally, their current approach to constitutional AI pretty much only makes sense if they believe that they can first teach the model what the Claude character is like and also teach the model that the persona responding is Claude, so I figure that has to be part of the pipeline even if they're not very good at it.
I'll be comparing the 9B vs Ling 3 Tiny (8B-A1B) as a scout model. Ling tiny is so fast but can be a little too dumb. Hope the 9B strikes a good middleground even if dense/slower.
ended up disabling ornith 9B. Oddly Ling 3 Tiny is pretty dang capable if its thinking is unleashed (tons of output tokens, maybe 5X the tokens but it's so fast it's maybe only twice as slow as a smarter model). This is a really interesting space if these super small models keep improving. SO FAST! Want to use!
Hoping this is real. It's too bad to see the signals from Qwen that they will not be releasing a 35B-A3B for the 3.8 lineup. The MoE architecture makes a huge difference for being able to run these local models on reasonable consumer hardware.
Honest question/suggestion for the HN audience- Since Qwen released the weights for Qwen3.8 2.4T-A95B and we already have the staring point of Qwen3.6 35B-A3B, couldn't someone distill the bigger model and make a "pseudo" Qwen3.8 35B-A3B? Sure, it wouldn't be an official Qwen release but couldn't someone improve on Qwen 3.6 and get the thing everyone is asking for?
I am calling this a suggestion for the audience because I don't have the will/resources to do this.
"Qwen3.8 35B-A3B" and 4B/9B variants are already on huggingface distilled by hobbyists.
>I can't find any. Do you have a link?
https://huggingface.co/Lord-H4D3ZS/Qwen3.8-Distill-35B-A3B-C...
> Base / architecture: Qwen/Qwen3.6-35B-A3B (Qwen3_5MoeForCausalLM, 256 experts, ~3B active). The "3.8" in the name refers to the teacher, not the base.
Not endorsement, haven't run it myself, just found the link.
[dead]
Yes, of course. But no one really wants to be the guy actually renting an entire B300.
we will have B300 Nodes available for off takers by Nov. please send inquiries to nmanney@edge-node.ai
..And there lies the problem.
The smarter 27B is so fast with MTP I've found I really don't need the 35B-A3B. You get around 70tk/s on a M5 Max lowering to around 40tk/s at higher context sizes.
I've benched 3.8 27B being significantly slower and less quality than 3.6 35B-A4B (both 4-bit quant, MTP, both using turboquant 4-bit served by oMLX), to the point that I'm not even using it right now (on an M3 Max). What's your use case and what did you observe? I might be missing something.
I believe you mean 35B-A3B, there was no such thing as A4B. I use 27B and other models for software development, and quite a few research or similar agents. I cannot imagine a world where the old 35B-A3B model is smarter / more capable than 3.8 27B - the difference is night and day for coding at least. Where 35B-A3B was fast and felt like a Haiku model, 27B feels like a strong Sonnet when given the right tools. I don't use turbo quant so can't comment on that, but with the A3B model you're using you probably won't get much from using MTP with small MoE models like that.
People over-quantize things, muck with the temperature and other settings based on superstitions or results from models they think are similar. There's lots of ways to make 3.8 27B dumber.
I've had great success with allowing the model to review optimal settings and my system specs. It comes up with the right configuration. Running Pi harness. I just had Ornith 1.5 take a moment to configure itself, now it's reviewing a large project I'm working on, so far its really impressive for my needs. Qwen3.8 27b Unsloth(Dynamic 3.0) is also perfect. These two are working together, and I'm in a sweet-spot, I now have all I need.
Yes, A3B, sorry for the typo. I’ve benched it and used it in practice for agentic coding, and I haven’t seen any benefit to it yet. Lightning MTP helps a lot in the perf department.
I would suggest careful benchmarking. I actually tested and benchmarked, and the new Qwen3.8-27B model is actually slower with MTP on my M4 Max. MTP only gains anything when generating long code sequences, which is very unlikely as the model spends most of its time thinking, not generating code, even if you use it for coding (which I don't).
I get 20 tokens/s on an M4 Max (larger GPU).
That shouldn't be the case, it sounds like you've got something else going on with your setup. Here's my benchmarks: https://omlx.ai/my/fadc2127d384283f5df1fcc2c093a9f95700c6a52... which are inline with the communities: https://omlx.ai/benchmarks/performance?sort=tg_tps&order=des...
I only get ~4 tok/sec on a M1 Pro with MTP.
I get ~10 tok/sec on a M1 Ultra, but the real issue is abysmal prompt processing which makes it unusable. On the M1 and M2 series MTP is actively harmful for performance.
Does 27b mean it fits one 32GB GPU?
27B means it has 27 billion parameters, but how that translates to model size depends on the model architecture. For Qwen 3.5 architecture (same as 3.8) the native size is around 55 GB memory. So you need quantization to fit it in 32 GB.
Go here: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
See the right hand side panel, you see a whole palette of quantizations and their respective sizes. Should give you an idea. Note that these are not the only quantizations available.
It doesn't mean that, but yes it would (with a 5 bit quant).
What quant, what runtime?
AWQ 5bit, oQ5. oMLX.
Quick tests suggest it’s pretty good at reasoning and tool use (keen to search to check its thinking) and it seems to waste much less time thinking, too.
So it feels very fast.
But it does not seem to be better than Qwen 3.6 35B at coding. A bit worse, I think, though I will test it more.
If you have a machine that can fit a 35B model in VRAM, I would suggest testing Muse Glimmer with (from memory)
in the system prompt.Despite being a dense model, this is actually capable of solving code problems faster than the Qwen MoE, despite having only one fifth of the raw token performance.
Personally I find speculative decoding much better strategy than MoE – performance wise it's there at 90-100 t/s on 2x4090, great intelligence – really great fit.
A lot of people, including me, don’t want to bother with GPUs, they’d rather run it on their M1-M5 MacBook. For example the 35B-A3B is very usable even on a M1 64GB MacBook.
Speculative decoding also works on Mac, 64G is more than what I have, m5 max should handle up to ~40 t/s with optimized setup (and with a lot of vram you can get great wins on concurrency – that harness can take advantage of for single user task as well), but agree memory bandwidth in mac or spark is still too slow, next gen for both will be great hardware to have for sure.
I get around 70tk/s on the m5 max, with 5bit AWQ / oQ5 slowing only to around 40tk/s at higher context.
MoE is great on systems that lack the VRAM to host the full model. On my 16GB VRAM system, I can get 100 tok/s with Q4 Qwen 3.6 35b a3b, and 15 tok/s with 27b.
MTP is a trade-off, as it pushes some more of the model off the GPU.
I have managed to get usable quants of Laguna S2 and even DeepSeek V4 flash on this setup.
There is clearly some intelligence loss compared to similar sized dense models, but I feel like it stomps on the 9-12b models I could run fully on GPU
> The MoE architecture makes a huge difference for being able to run these local models on reasonable consumer hardware
That's not true. For computers without unified memory architecture (which is the vast majority) VRAM capacity is the bottleneck for local models. In that case a dense model can deliver significantly more intelligence than an MoE model of the same size. And for a typical consumer/gamer Nvidia GPU, dense models are fast enough.
Expert offloading significantly helps with the VRAM capacity.
Most MoE architectures have a few experts that are always running; this, the router, KV, and whatever else you have space for can stay in fast VRAM; and the remaining experts can be offloaded.
llama.cpp can run MoE with some layers in vram and some layers in ram
I've been experimenting with it on a M4 Pro 24G for the last few hours and it's been very promising using 32k context. getting around 30-40 tps
With Qwen3.8 27B I could not get anywhere near 32k context window, that made it very unusable for agentic coding, although it was very smart.
I'm running qwen3.8 27B dense on reasonable hardware (oem spark)
tbh, I have stopped using MoE in the name of speed, the dense (with more active parameters) makes a real difference in output quality
The way it was said made it sound like they had something better than a 35B-A3B coming.
What signals? Looking more like a tiered release.
This is all real. More real things coming soon.
Apparently this is Jiwei Li's new company. https://ai.miraheze.org/wiki/Ornith https://www.innovatorsunder35.com/the-list/jiwei-li-2020/
I wonder what their angle is going to be; the scene is crowded, and they don't do serving.
Can't wait to try this. Ornith1 (9B) was a really nice model. I have been running it locally using - https://github.com/deepanwadhwa/samosa-chat
What did you use it for? I've found it somewhat capable but not worth to actually use it (1 9B, that is)
How is ornith-1.5's base model developed? Is the base model one of the Open weights models, or one pre trained by ornith team from scratch? I couldn't find information to answer this question in the article.
It looks like they post-trained Qwen3.6. Interesting to see how far they could improve it with they harness/algorithm.
I've been using the 35B-A3B today for some web scraping work, and it has been on par with Qwen3.8 27B at a much higher speed and at a higher quant (q4 vs q8). I'm impressed.
What you say is in agreement with someone I follow on X. My family has been traveling for two weeks and I have been looking forward to trying qwen3.8-37b when I get home.
Now I might try Ornith-1.5-35B first even though I don’t see an official MLX version.
I had to go down to UD-Q3_K_XL for Qwen 3.8 27B to get it to fit in VRAM and be usable, but I worry I'm gutting its intelligence somewhat. I too am interested in faster + more-usable alternative that can exchange blows with the Q3-dumbed 27B.
I need someone to run actual benchmarks between the two.
Benchmarks are the BMI of model evaluation.
They may have utility in trying to look at the whole landscape of models, but are very misleading when it comes to making 1:1 comparisons or in developing confidence at to how a given model will deliver on your workflow.
> Benchmarks are the BMI of model evaluation.
That is such an elegant way to put it.
Only relevant benchmarks are those you make yourself, targeted specifically for your workflows. Anything else is just number go up on a pretty graph, and every model out there is probably benchmaxxed to hell on the public ones anyway. Keep yours private.
These models have gotten a fair amount of attention -- we're hoping it's enough to get them added to some reliable inference providers and OpenRouter, at which point we'll run them on our full benchmark suite.
The page has comparisons with Qwen 3.6 27b and I’d love to see comparisons with Qwen 3.8 27b, the newer one is much more capable!
It's somewhat close, but a lot worse at code it seems. Qwen 3.8 is a wild improvement over 3.6.
Interestingly, in my own benchmark and testing (in the hopes of finding a good-enough local model to run a personal assistant agent), Ornith-1.0-9B was worse than Qwen3.5-9B which according to their scores should've been reversed.
I will definitely pass Ornith-1.5-9B through the gauntlet as well!
Please check your download links, it get 400 Bad Request on https://huggingface.co/ornith-ai/Ornith-1.5-397B-GGUF/tree/m...
only mmproj can be downloaded right now
397 is just too big for two Sparks even at NVFP4. Wish they had made this just a tiny bit smaller.
Ornith-1.5-397B is derived from Qwen3.5-397B-A17B via post-training. That process preserves exact parameter count.
They should've included Qwen3.5-397B-A17B in the benches then :/
Ah thanks for the background. Well, time to learn how to quant things down!
Can someone parse all that AI generated blather in the post and tell me clearly:
1. Is this self improvement at the model level (updated weights or memory, KV etc) or just by adding agentic code harnesess to guide the output better?
Thank you
The former
This is exciting.
Their 9B model benchmarks competitively with Sonnet 4 which is pretty cool to have such a small model compared to one that came out 10 months ago.
I’m curious how providers will price their 397B model.
It’s time for me to upgrade the main server in my home lab and I’m thinking about which machine should I have.
What kind of hardware you’d need to run the 397B one at an acceptable speed?
3 DGX Sparks with 400 Gb interconnects in a loop.
I was looking for something more general-purpose, such as a decommissioned quad-socket x86 with 80 or more AVX512 cores and a terabyte of RAM, which I suspect will be a strong limiting factor. The possibility of adding some older Nvidia accelerators is interesting as well even though they won’t be able to hold the whole model in HBM and will need to stream it over PCIe 4, which sucks.
That way, when not running inference, the machine can easily host VMs for other experiments and general housekeeping functions.
What fraction of GPU-hours in each cycle went to rollout generation versus the actual update?
Is this open weights? Or planned to be
Yes: https://huggingface.co/collections/ornith-ai/ornith-15
Across five cases it reliably claims to be Claude without being able to name a specific version.
Nobody (with the probable exception of Anthropic given their work on character training) really trains models on their identity and Claude is the only AI persona that's well-defined so if you put yourself into the AI's shoes it's a pretty reasonable guess that it might be Claude. I've had basically every open model claim it's Claude when the topic comes up.
Anthropic changed their character training policy many times, and the name is probably separate from it anyway (besides the bits from the constitution etc). Name training usually comes last in all models, if at all, and it's pretty shallow. Certain Claude models say they are Qwen or Deepseek when asked in Chinese, for example.
Yeah, Claude is actually surprisingly unsure of his identity considering that their most recent publication on their constitutional AI training literally had graphs demonstrating how certain properties differed based on whether they were phrased as questions about "Claude" versus "You", but it's actually that which makes me fairly confident that they're probably doing _something_ to try and close that gap.
More generally, their current approach to constitutional AI pretty much only makes sense if they believe that they can first teach the model what the Claude character is like and also teach the model that the persona responding is Claude, so I figure that has to be part of the pipeline even if they're not very good at it.
I'll be comparing the 9B vs Ling 3 Tiny (8B-A1B) as a scout model. Ling tiny is so fast but can be a little too dumb. Hope the 9B strikes a good middleground even if dense/slower.
ended up disabling ornith 9B. Oddly Ling 3 Tiny is pretty dang capable if its thinking is unleashed (tons of output tokens, maybe 5X the tokens but it's so fast it's maybe only twice as slow as a smarter model). This is a really interesting space if these super small models keep improving. SO FAST! Want to use!
This looks cool
[flagged]
Another day another startup claiming some vague version of RSI to try to close their round.