Recently I had switched to OpenCode to try out many of the Non-US-Frontier-Labs models. My unexpected favorite model to use was Mercury (a diffusion model). Not because it was “smart” but because it was stupid fast. It was more of a pair-programming experience instead of the SOTA agentic experience of prompting and waiting. Honestly, it was also way more fun and brought back some of the pre-AI coding experience while still getting some benefits of AI. It felt less of a slot machine where you prompt, wait, and hope it went in the right direction. It made me even use the tiny models like Gemini Flash Lite and GPT Mini/Nano more too.
Anyways, so excited for an open-weight model and I hope it performs well. I’ll be testing this ASAP.
If you can run your tests fast and cheaply, and have metrics that show what bad/sloppy code is that are cheap & fast to generate, a worse fast model can outperform a far better far slower model if you value time...
I've had pretty good success with LLMs after putting in place metrics to measure true complexity (not cyclomatic), and automatically pushing back everything until the added complexity is within reason for the feature.
There's a ton of research on this in the 80s... and interestingly, I haven't seen a lot of recent research.
Surprisingly, it seems most languages don't have a standard package to do a lot of these detections.
Ruby has Flay to detect similarity (something LLMs are prone to do). Basically re-write a huge function with only a couple of minor differences that should probably be params...
One of the things I rely on most is "pressure" -> which conditions are causing the most checks throughout the code-base. Those are things you should Type away.
Dynamically typed languages like Ruby create a huge surface area for type slop for LLMs, and why I would not recommend using a dynamically typed language for vibe coding.
You can have type "pressure" and nil "pressure" -> where you set a value to nil somewhere (that you probably shouldn't have) -> and that has ripple effects all throughout your codebase. Similarly, you can do this for values -> one place it's a string (where it shouldn't be), everywhere else a symbol (what it should be) -> but now you've got hundreds of casts to_sym or to_s in your codebase.
There's also state drift & reification misses -> you constantly update two states (that should probably just be one new value or a function) and sometimes you forget to update one (more of a bug possibility than complexity). Same for reification misses -> you constantly check for multiple conditions -> that should probably be one value or a function, and similarly (buggy, you may sometimes miss one).
Complexity comes down to state and control flow -> so you want to check what's causing you to make the most decisions (especially state/time based), and where it's coming from. Where do you have the most state and why...
I'm hoping to release everything in the next few weeks, but it takes a while to polish things, especially when it's a side-quest of a side project...
> Dynamically typed languages like Ruby create a huge surface area for type slop for LLMs, and why I would not recommend using a dynamically typed language for vibe coding.
I totally understand this, and have seen the problems firsthand. But Elixir / Phoenix / LiveView, along with Tidewave, have become my favorite "vibe slop stack." Just so quick and easy, and the LLM seems to get things right quite often.
I wonder if a dedicated client or mode in a client would provide some benefits. Might also be interesting to do adversarial stuff too where it argues with itself or another model
I wonder how much this will impact locally used models for coding. I can imagine using diffusion models that are x-times faster than Qwen or Gemma 4 - where I have to do more "pre-ai" work which is a good thing and can have a very fast, very cheap model to work with locally. I assume since it doesn't do heavy computing for a long time that it's cheaper to run on local hardware as well?
Mercury-2 is amazing. I am using it frequently as the arbiter in llm-consortium
The context window is relatively small, so to make it work with larger consortiums I can construct a recursive sort-of meta consortium like this:
I've found the average output of many suboptimal models is still suboptimal, especially when it comes to judging the accuracy/correctness of the work of other models.
I did some benchmarks recently of how well various models find security vulnerabilities, and then follow up testing of the judging process of whether the models found the right bug and whether other bugs it reported were false positives or legitimate other bugs. A committee of good-not-great models (DeepSeek, MiMo, Gemma 4) cannot replicate the accuracy of Opus by itself. Even when all three of the other models disagreed with Opus, Opus was almost always the one that was actually right.
It's an interesting area for research. And, a model that's very fast can make a lot more attempts at a solution, and in cases where there is an unambiguous "right" solution that can be proven by some sort of static rule, "very fast" may be a useful characteristic. Small classification problems, where you need to make thousands of decisions about some specific aspect of a large corpus of data, seems like a sweet spot for a model like Mercury.
I have had a better experience with my own use. I use it every day and it rarely fails to improve tasks. Perhaps the prompts and rubrics make a difference. And finding bugs is one of the better use cases because it is essentially a search problem. As long as models are non-deterministic and there is some diversity in training data, then an ensemble that iterates on the problem is more likely to cover the ground needed to find solve a problem.
Some tasks benefit from this approach more than others. There was a paper from google on a version they made which was very similar and achieved SOTA then on planning and pathfinding benchmarks.
I get exactly what you mean. After getting frustrated with how slow Claude was on my personal projects, I switched to Google Antigravity with Flash models and the speed difference is huge. I feel more in the flow and just more focused on the task. I did not realize how much a difference speed can make.
Claude is better for extremely complicated, large codebases where its slower response time might be a good trade-off for the complexity of the task. Antigravity and other fast models works so much better for smaller projects where you want a "flowy" code, run, debug cycle.
YESSSS!!! speed is THE way! I like my boilerplate POJOs/data classes generated at breakneck pace of 300+ tok/s, Flash-Lite is more useful than GPT-5.5 for me this way. if it's too slow, you just stay in that goddamn async death loop
> I like my boilerplate POJOs/data classes generated at breakneck pace of 300+ tok/s
Regardless of speed, use the LLM to eliminate the need for boilerplate rather than just creating more code faster.
> if it's too slow, you just stay in that goddamn async death loop
Things get slow when you're ballooning the size of your code, files, design and architecture, and things get more involved and complicated, piling fast hacks on top of fast hacks and everything get brittle.
previously, mugging through docs to turn them into serializables for some API took weeks of grueling work if you wanted to cover an entire API surface that's as big as, say, GitHub's. nowadays, just "copy Markdown" from the very same GitHub, put 10-12 data classes, and let LLM extrapolate from there. with Gemini's 65.5k max token output, that is just several prompts and about two hours. that's the boilerplate. there is practically no way to automate this unless GitHub adopts OpenAPI spec in a way that's not buggy, so that we can just hit an endpoint and point procedural source generators at them
Sounds like you're trying to just re-implement a HTTP API, not really boilerplate.
Boilerplate is code you could have avoided written, but you take the "temporary" shortcut of copy-pasting the code instead of building a proper abstraction. This is what I'm talking about is the wrong direction.
For boilerplate, yeah. But when asking research or exploratory questions, or weighing whether a feature is well designed, or asking "can I implement _x_ feature using these libraries without introducing unnecessary complexity", then GPT-5.5 medium is still fast enough.
10-20 seconds times a couple turns on a new feature isn't bad. Kimi is also similarly fast if not faster.
I do agree with smaller models for more constrained/routine tasks though.
well, I can usually think for myself or hit someone up in Discord (or Teams, if it's for a living) and in a worst case (that person just deflects to AI anyway) just save some token budget for myself
I always think for myself too, but when learning to do something I've never implemented before, it's nice to have little sanity checks using something with the reasoning ability (plus the fast natural language search on hundreds of pages of documentation) of a model like GPT-5.5.
Every line I put in my app, I still reason about myself. But when deciding between 5+ ways of building some random, non-straightforward feature, it's nice to have what's essentially a "mentor" AI.
Imagine you’re entirely pre-AI… to do some work, you read code, think, then write some code across a number of files. Usually then a small dance with compilation/unit tests to address anything broken. Along the way, you use your human judgement on style and quality, and midway through your change you might refactor something based on learned best practices (eg, when to use a static method, or helper class).
Today, even the dumbest AI agents can trivially loop through the final dance to get compilation, and often unit tests (depending on scope of failure). Big SOTA agents have OK code quality, but if left unattended or unchecked will still generate pretty sloppy repos after a while. That’s true even when using models like Opus which is ridiculously expensive in comparison.
When using the models in this fast “pair programming” style, I find that I (the human) mostly do all the “plan and think” work, and usually point the smaller agent towards specific files/directories, with specific targeted changes. It’s slower than 1-shot prompting an entire feature, but slightly faster than doing it manually, and I find the code is less “slop” because the changes are smaller and more human. It retains the agentic benefits of handing imports, compilation iteration, etc and can do basic cross-file plumbing. It’s also cheap and fast to do iterations like “wait make that method static” or “let’s update this to use <other util class>” and things like that. When the agent is slow to make localized edits, I find I’m less likely to push for minor nit-picks and style updates.
Some of these comments miss the advantage of diffusion. This is will have a big impact on edge devices, such as your phone or the GPU in your computer.
An LLM's decoder computes tokens one-at-a-time because attention has to account for each previous token. The existing LLM decoders scale well when you have enough load to batch many inferences together. Diffusion of limited benefit there. On edge you have a different problem: your inference accelerator is starved while sloshing GB of weights back and forth from RAM. That's because the consumer RAM like LPDDRx/GDDRx is lower bandwidth than HBM, and the requests are serial so you can't batch compute common weights. Diffusion can compute tokens in parallel which relieves the memory bandwidth bottle neck.
Edge devices don't just have limited memory bandwidth though, they also have very limited compute. To the extent where you don't actually need all that much batching to saturate their viable compute and run into obvious thermal/power limits. (It's just not true that "requests are inherently serial" in edge inference; any time you have multiple requests (i.e. "chats") in flight, batching becomes applicable if you have enough memory capacity for the KV caches.) I'm not sure how diffusion models are supposed to help there, if they simply take more compute for lower-quality outcomes and a dubious saving in memory bandwidth.
Forgot to mention it previously, but this might be a good model for a narrow slice of midrange systems that really are more skewed towards compute than memory bandwidth, but also don't have enough memory capacity to effectively use batching. (E.g. top-of-the-range consumer GPUs, or earlier generations of datacenter GPUs.) Although you do also compete with things like MTP there, which is targeting a similar tradeoff, or with denser models featuring a similar amount of total parameters. So I'd say that the jury is very much still out, even in that narrow space. Diffusion models are also apparently very hard to scale to a hundred-billion or trillion parameter count, since the way you train them is completely different to the usual one-token-at-a-time models.
Google keeps flexin'. It's surprising that Gemini isn't more competitive against Claude or OpenAI models for code and agentic use, because it's clear Google still has some of the best AI people in the business. But, I guess Google is focused on stuff that runs on phones and near-realtime use cases, rather than the big thinky LLMs.
All these efficiency improvements seem likely to be really important to the future of AI, though, as the money starts flowing the other direction. The days of subsidized tokens to try to lock people into specific ecosystems are coming to an end, and we're going to have to start paying what it actually costs.
The companies that figure out how to make it cost-effective to run really smart models are the ones that will win. DeepSeek costs an order of magnitude less than GPT 5.5 or Opus 4.8. It's worse than either, but not catastrophically worse. I'll happily pay ten times as much for the best coding model, because it saves enough human time to justify it, but not a hundred times as much, which is where things seem to be heading (GPT 5.5 Pro cost over 200 times as much as DeepSeek in some benchmarks I recently did, and ~30 times as much as Opus 4.8).
Google is clearly gimping the gemma models. There is a 122b gemma 4 that was never released, but was a part of the announcement tweet. Plus they weren't going to release MTP until people figured out they're running it on the pixels
I dunno about that. Gemma 4 is probably the best model for general self-hosted use for almost everyone that doesn't have a data center in their basement. They didn't have to release it at all, and they didn't have to release speculative decoding drafters, and they didn't have to release the QAT version of the models that makes the 4-bit quantization perform very close to the bigger versions, and can run in 32GB. I'd love a 122B version of it, and I didn't realize they'd ever announced one was coming (though I remember there being speculation about it). But, also, I'm happy they're doing so much with it. They've got all the sizes covered, it has great prose for an LLM, better prose than even most larger models, it's got great audio and vision, and broad language support. As self-hosted general purpose models go, it's the total package.
Qwen 3.6 is maybe better for code (though I'm beginning to think otherwise after some benchmarking I've been doing, where Gemma 4 has been overperforming expectations), but for just about anything else, Gemma 4 is the one.
If they're gimping it, why is nobody else making a better one that small?
Fable's costs are twice Opus' and it's clearly quite competitive with GPT-Pro, so that seems like it might be a good option for you if the trigger-happy safeguards aren't too much of a problem. Google has their own "Deep Research" option in this space which seems to work well.
The nice thing about DeepSeek is its ability to be run on local hardware, with no API costs involved. If you care deeply about that, then it being a bit worse than Opus or GPT isn't really a problem.
I think Google will win out in the end. They are concentrating on what matters, performance per watt, and performance per dollar. They are building their own inference hardware and are working towards edge-computing which removes latency and compute overheads. These big LLMs are not yet cost effective, Google is just letting them burn their investment funds to "sell" to consumers at below cost.
After the AI bubble bursts, it will be the likes of Google that come out the other side still wearing their shirts. I think this bubble is out to scalp some giants.
Maybe with very fast models you could request animation frames, e.g., frame 1) right foot at 12, left foot at 6; frame 2) right foot at 3, left foot at 9, etc.?
And instead of reporting tps, you would - of course! - report pfps (pelican frames per second).
Thought of you this afternoon, "after you click the record button can you make a 'boop, boop, boop, clack!' like a lead-in from a from a clapboard (using web audio synthesis apis)?"
> DiffusionGemma reverses this inefficiency. Instead of predicting words sequentially, it drafts an entire 256-token paragraph simultaneously. By giving the computer's processor a larger chunk of work at once, DiffusionGemma utilizes your hardware to its full potential. It upgrades your model inference from a single, sequential typewriter to a massive printing press that stamps the entire block of text simultaneously.
> Operating as a 26B total Mixture of Experts (MoE) model that activates only 3.8B parameters during inference, DiffusionGemma fits comfortably within 18GB VRAM limits of high-end dedicated consumer GPUs when quantized.
Okay, so Gemma 4 26B is a MoE model that's really fast on my 24 GB GPU using ollama. This sounds like speculative decoding but I don't think that works with MoE models? It's hard to keep up with all this when it's not your job to keep up with it.
This is a different model with, confusingly, approximately the same number of params as the existing gemma4 MoE. Unclear from a quick scan whether one was trained somehow from the other.
The mechanism isn't the same as speculative decoding. Speculative decoding happens sequentially and (usually) a couple of tokens at a time; diffusion doesn't, and does blocks of text at once. I haven't read the collateral yet but my assumption would be that it's trained to keep the specific experts stable across a diffusion block.
A few days ago I was just thinking that Google never talked about their diffusion text generation model after demoing it at I/O a year ago. The rumor is that it was too expensive to run, but with the provided chart using the same 1x H100 hardware and comparing DiffusionGemma to regular Gemma, that shouldn't be the case. I'm curious what the downside for this speed is here aside from being slightly weaker than Gemma.
> I'm curious what the downside for this speed is here
"DiffusionGemma's speedup is designed for local and low-concurrency inference. In high-QPS cloud serving, autoregressive models can be deployed to saturate compute efficiently, so DiffusionGemma's parallel decoding offers diminishing returns and can result in higher serving costs"
Well with a standard autoregressive model you can generate for example 256 tokens at once if you have 256 users, with this approach you can generate 256 tokens for a single user but you need several forward steps.
So the diffusion process takes more GFLOPs, if you have enough users you can already balance memory and compute.
It is cool but local models while okay already feel noticeably worse than even the cheapest APIs so I can't see myself sacrificing even a little bit of their quality for speed. I'm sure it's worth it for some usecases, curious to hear specific ones that people are already planning to deploy to production.
I can see it but even if I do that for something like tests I'd still eat the time cost of the normal Gemma for 10% extra performance. And further, if you switch between the fast and normal Gemma for different tasks you eat the big time cost of loading the other model (and maintaining both in the first place).
What would a diffusing reasoning model look like? have a pre-defined length [thinking] block that gets diffused over a long time, and then the final output block uses what is in that thinking block as part of its input?
And how do diffusion models decide the output length in the first place, is it a pre-set parameter? or does it diffuse an [end] token into the middle somewhere?
MTP is a training optimization. Drafting requires verification, and verification is the full model inference. Speculative decoders are the name for the inference time optimization, that is more like a verifier that is a smaller model.
Has anyone evaluated any diffusion LLMs for error spotting?
E.g. run your normal autoregressive LLMs (with MTP whatever, as you like), then run a single diffusion pass over the result, and observe any tokens that diffusion thinks are unlikely.
Then prompt the autoregressive llm with some structured reasoning "<think>Is <diffusion unlikely part> an error? .."
Because the diffusion model is so structurally different perhaps it makes different errors such that this would provide gains even vs running distinct autoregressive LLMs which often make the same errors.
The same argument could apply for RWKV but it would be relatively expensive to apply it as a second pass on a big block of output, while it seems like a diffusion model would be cheaper.
Compared to autoregressive decoding, diffusion is huge for local MoE inference because of the improved token generation efficiency, especially for normal GPU + ram offload setting.
However, there are models which are better positioned on the performance vs memory pareto front, i.e. dense models, so I'll just wait.
P.S. QAT is really something as it reduces the performance fluctuations compared to the normal one. Thanks again.
I'm not getting anywhere near the speeds advertised on my 3090 Ti, alas, but it's fun watching it "fill out" its answers. I did Simon's "SVG pelican on a bicycle" test on it and the result was quite minimalistic but fit the brief: https://gist.github.com/peterc/7672e74ec1437945e5fca5ce2c1c9... -- this was on the Q4 quant running on patched llama.cpp. I will be interested to see if Simon's looks much different.
Yeah, the patched llama.cpp. The reason is I saw that using the Q4 quant on vLLM is discouraged and the int8 won't fit on my 3090 Ti, but I could certainly give it a go. I also skipped Transformers as it needs to download the full weights and quantize them locally and I didn't fancy waiting for a 50GB download.
Maybe someone can explain: in image generation some models are already using rectified flow. Which was hailed as the next big thing. Are we going to see discrete rectified flow models next or is that unlikely?
The intrinsic limitation of text diffusion is that natural text contains serial dependencies where a word at the beginning of the text strongly influences what comes later, and if there is a long enough dependency chain within a diffusion block, the small number of diffusion steps may not be enough to resolve all dependencies, so that you end up with incoherent output.
I'm curious how diffusion models do at tool calling, curious what wins there are there.
The video demo of the svg sword is an interesting example of what is so interesting about diffusion models: it's not just putting one token after another to make edits to a file. It's skipping around, it's re-editing previous lines. I feel like forcing it to write too calls is maybe not its best nature.
I feel like perhaps instead of a monolithic edit file tool call, perhaps the diffusion model would be better suited to posting a change stream, a series of edit ops, across multiple files.
Almost certainly not if things remain as they are. The reason there's been little traction is the quality gap between diffusion and autoregressive models is pretty stark. I mean just look at the benchmarks here. Large dropoffs, with the hardest benchmarks seeing the largest drops. On top of that, almost all the speed benefits of diffusion models become negated at scale. So this is only attractive for local model development and almost everyone training local models still care about pound for pound quality and inference efficiency at scale.
It's fast enough that "ask it twice and pick the best" should still come out ahead performance-wise. I don't know how much that would close the quality gap by, but it's worth a play.
The thing is, diffusion models perform somewhat worse than autoregressive on text. So you lose some performance.
Speed is the big advantage. Autoregressive when doing local inference is mostly memory bound; you're doing one token at a time, for each token you need to load all weights. MTP helps a bit by allowing you to draft tokens in a smaller model and then verify them in parallel with the larger model, allowing you to do a few computations for every memory load, but because you're still doing tokens sequentially and need to discard invalid drafted tokens, you can only get so much speedup.
For hosted models, however, you can batch many token generations together, fully utilizing all of the compute while no longer being bottlenecked on memory bandwidth. So they are already operating at close to max efficiency.
So, diffusion kind of loses its beneifit in hosted models. Sure, maybe you could pay more to have slightly lower latency responses by doing diffusion for one user at a time instead of autoregressive for many in parallel. But given that it also reduces accuracy, it's hard to see where you'd really want that. Unless they're able to bring it up to par with autoregressive, it seems like it's a bit of a dead out outside of local models where you're generally just doing one thing at a time.
I'm particularly curious to know how this plays out, and I seriously hope that more labs focus on diffusion models for text usage.
My immediate thought - this performs slightly worse than the autoregressive gemma equivalent, but it may also let me functionally run better models in diffusion variants.
Ex - I can run 70b-120b autoregressive models locally right now, but I get ~5-15t/s, which just isn't fast enough for serious work.
Which caps me down in the 20-36b models (ex - gemma4) where I can get 100+t/s on the same hardware.
So the question becomes - does the quality drop from a diffusion model outweigh the quality bump from using a larger model?
Because if not... sounds like diffusion models have a lot of space to thrive.
---
Sadly - if they can't be hosted profitably, I question whether this space will actually be explored.
it just me that thinks its kinda weird that they conflate speed in tokens/second and latency, when i think of latency as time to first token? like it generates an entire paragraph of tokens faster but wouldnt it still be slower if your reply is only 1 word because it has to do the entire 256 tokens as a chunk
I wish labs would do QAT and release these quants, at this point looking at releases of bf16 without QAT feels like looking at half backed bread, we can quantized it but it is not the same as QAT. Or I am missing something here ?
I can’t help but feel like there’s something here that will matter for future LLMs.
The bidirectionality could be a big deal: being able to refine a sentence with both left and right context feels closer to how editing/thinking actually works than committing to each token forever.
Maybe the current models aren’t good enough yet, but the direction feels important.
We need more local open weight models that are performant and just as good (or good enough) as the best frontier ones.
Then you will be able to achieve Jevons Paradox and enjoy the same “productivity gains” without paying for these extortionate token prices by closed model providers or have it as cheap as possible.
We have this though, right? Compare SOTA local models to where the frontier was last year. There weren't many people complaining that last year's frontier models were incapable.
Next year, and the year after, Fable, GPT 5.5 and Gemini 3.5 will feel quite ordinary. And perhaps even within reach of a prosumer running models locally.
Recently I had switched to OpenCode to try out many of the Non-US-Frontier-Labs models. My unexpected favorite model to use was Mercury (a diffusion model). Not because it was “smart” but because it was stupid fast. It was more of a pair-programming experience instead of the SOTA agentic experience of prompting and waiting. Honestly, it was also way more fun and brought back some of the pre-AI coding experience while still getting some benefits of AI. It felt less of a slot machine where you prompt, wait, and hope it went in the right direction. It made me even use the tiny models like Gemini Flash Lite and GPT Mini/Nano more too.
Anyways, so excited for an open-weight model and I hope it performs well. I’ll be testing this ASAP.
If you can run your tests fast and cheaply, and have metrics that show what bad/sloppy code is that are cheap & fast to generate, a worse fast model can outperform a far better far slower model if you value time...
I've had pretty good success with LLMs after putting in place metrics to measure true complexity (not cyclomatic), and automatically pushing back everything until the added complexity is within reason for the feature.
How do you measure “true” complexity? Cyclomatic seems a bit… I dunno, artificial? Blunt? But it has the benefit of being defined.
There's a ton of research on this in the 80s... and interestingly, I haven't seen a lot of recent research.
Surprisingly, it seems most languages don't have a standard package to do a lot of these detections.
Ruby has Flay to detect similarity (something LLMs are prone to do). Basically re-write a huge function with only a couple of minor differences that should probably be params...
One of the things I rely on most is "pressure" -> which conditions are causing the most checks throughout the code-base. Those are things you should Type away.
Dynamically typed languages like Ruby create a huge surface area for type slop for LLMs, and why I would not recommend using a dynamically typed language for vibe coding.
You can have type "pressure" and nil "pressure" -> where you set a value to nil somewhere (that you probably shouldn't have) -> and that has ripple effects all throughout your codebase. Similarly, you can do this for values -> one place it's a string (where it shouldn't be), everywhere else a symbol (what it should be) -> but now you've got hundreds of casts to_sym or to_s in your codebase.
There's also state drift & reification misses -> you constantly update two states (that should probably just be one new value or a function) and sometimes you forget to update one (more of a bug possibility than complexity). Same for reification misses -> you constantly check for multiple conditions -> that should probably be one value or a function, and similarly (buggy, you may sometimes miss one).
Complexity comes down to state and control flow -> so you want to check what's causing you to make the most decisions (especially state/time based), and where it's coming from. Where do you have the most state and why...
I'm hoping to release everything in the next few weeks, but it takes a while to polish things, especially when it's a side-quest of a side project...
Interesting, I do think blending the fuzziness of models with the determinism of hard checks/conformance is the way too go.
But using some kind of metrics as guardrails/steering seems interesting.
> Dynamically typed languages like Ruby create a huge surface area for type slop for LLMs, and why I would not recommend using a dynamically typed language for vibe coding.
I totally understand this, and have seen the problems firsthand. But Elixir / Phoenix / LiveView, along with Tidewave, have become my favorite "vibe slop stack." Just so quick and easy, and the LLM seems to get things right quite often.
I wonder if a dedicated client or mode in a client would provide some benefits. Might also be interesting to do adversarial stuff too where it argues with itself or another model
What metrics have you found useful?
I wonder how much this will impact locally used models for coding. I can imagine using diffusion models that are x-times faster than Qwen or Gemma 4 - where I have to do more "pre-ai" work which is a good thing and can have a very fast, very cheap model to work with locally. I assume since it doesn't do heavy computing for a long time that it's cheaper to run on local hardware as well?
Mercury-2 is amazing. I am using it frequently as the arbiter in llm-consortium The context window is relatively small, so to make it work with larger consortiums I can construct a recursive sort-of meta consortium like this:
Now when I prompt cns-meta-glm-kimi it will pick the best of five from kimi and glm before creating a synthesis from the two winners.I've found the average output of many suboptimal models is still suboptimal, especially when it comes to judging the accuracy/correctness of the work of other models.
I did some benchmarks recently of how well various models find security vulnerabilities, and then follow up testing of the judging process of whether the models found the right bug and whether other bugs it reported were false positives or legitimate other bugs. A committee of good-not-great models (DeepSeek, MiMo, Gemma 4) cannot replicate the accuracy of Opus by itself. Even when all three of the other models disagreed with Opus, Opus was almost always the one that was actually right.
It's an interesting area for research. And, a model that's very fast can make a lot more attempts at a solution, and in cases where there is an unambiguous "right" solution that can be proven by some sort of static rule, "very fast" may be a useful characteristic. Small classification problems, where you need to make thousands of decisions about some specific aspect of a large corpus of data, seems like a sweet spot for a model like Mercury.
I have had a better experience with my own use. I use it every day and it rarely fails to improve tasks. Perhaps the prompts and rubrics make a difference. And finding bugs is one of the better use cases because it is essentially a search problem. As long as models are non-deterministic and there is some diversity in training data, then an ensemble that iterates on the problem is more likely to cover the ground needed to find solve a problem.
Some tasks benefit from this approach more than others. There was a paper from google on a version they made which was very similar and achieved SOTA then on planning and pathfinding benchmarks.
edit:
Mind Evolution paper https://deepmind.google/research/publications/122391/
(That was a month after I published llm-consortium :) https://xcancel.com/karpathy/status/1870692546969735361
I get exactly what you mean. After getting frustrated with how slow Claude was on my personal projects, I switched to Google Antigravity with Flash models and the speed difference is huge. I feel more in the flow and just more focused on the task. I did not realize how much a difference speed can make.
Claude is better for extremely complicated, large codebases where its slower response time might be a good trade-off for the complexity of the task. Antigravity and other fast models works so much better for smaller projects where you want a "flowy" code, run, debug cycle.
YESSSS!!! speed is THE way! I like my boilerplate POJOs/data classes generated at breakneck pace of 300+ tok/s, Flash-Lite is more useful than GPT-5.5 for me this way. if it's too slow, you just stay in that goddamn async death loop
> I like my boilerplate POJOs/data classes generated at breakneck pace of 300+ tok/s
Regardless of speed, use the LLM to eliminate the need for boilerplate rather than just creating more code faster.
> if it's too slow, you just stay in that goddamn async death loop
Things get slow when you're ballooning the size of your code, files, design and architecture, and things get more involved and complicated, piling fast hacks on top of fast hacks and everything get brittle.
Slow is fast, longer-term anyways.
previously, mugging through docs to turn them into serializables for some API took weeks of grueling work if you wanted to cover an entire API surface that's as big as, say, GitHub's. nowadays, just "copy Markdown" from the very same GitHub, put 10-12 data classes, and let LLM extrapolate from there. with Gemini's 65.5k max token output, that is just several prompts and about two hours. that's the boilerplate. there is practically no way to automate this unless GitHub adopts OpenAPI spec in a way that's not buggy, so that we can just hit an endpoint and point procedural source generators at them
> that's the boilerplate
Sounds like you're trying to just re-implement a HTTP API, not really boilerplate.
Boilerplate is code you could have avoided written, but you take the "temporary" shortcut of copy-pasting the code instead of building a proper abstraction. This is what I'm talking about is the wrong direction.
For boilerplate, yeah. But when asking research or exploratory questions, or weighing whether a feature is well designed, or asking "can I implement _x_ feature using these libraries without introducing unnecessary complexity", then GPT-5.5 medium is still fast enough.
10-20 seconds times a couple turns on a new feature isn't bad. Kimi is also similarly fast if not faster.
I do agree with smaller models for more constrained/routine tasks though.
well, I can usually think for myself or hit someone up in Discord (or Teams, if it's for a living) and in a worst case (that person just deflects to AI anyway) just save some token budget for myself
I always think for myself too, but when learning to do something I've never implemented before, it's nice to have little sanity checks using something with the reasoning ability (plus the fast natural language search on hundreds of pages of documentation) of a model like GPT-5.5.
Every line I put in my app, I still reason about myself. But when deciding between 5+ ways of building some random, non-straightforward feature, it's nice to have what's essentially a "mentor" AI.
Mercury is a US LLM from https://www.inceptionlabs.ai/
Wow... I forgot about that. Mercury is brutal. I had him review lint errors and the speed is just insane
Could you say more about how you use it? What does your workflow look like?
Imagine you’re entirely pre-AI… to do some work, you read code, think, then write some code across a number of files. Usually then a small dance with compilation/unit tests to address anything broken. Along the way, you use your human judgement on style and quality, and midway through your change you might refactor something based on learned best practices (eg, when to use a static method, or helper class).
Today, even the dumbest AI agents can trivially loop through the final dance to get compilation, and often unit tests (depending on scope of failure). Big SOTA agents have OK code quality, but if left unattended or unchecked will still generate pretty sloppy repos after a while. That’s true even when using models like Opus which is ridiculously expensive in comparison.
When using the models in this fast “pair programming” style, I find that I (the human) mostly do all the “plan and think” work, and usually point the smaller agent towards specific files/directories, with specific targeted changes. It’s slower than 1-shot prompting an entire feature, but slightly faster than doing it manually, and I find the code is less “slop” because the changes are smaller and more human. It retains the agentic benefits of handing imports, compilation iteration, etc and can do basic cross-file plumbing. It’s also cheap and fast to do iterations like “wait make that method static” or “let’s update this to use <other util class>” and things like that. When the agent is slow to make localized edits, I find I’m less likely to push for minor nit-picks and style updates.
So you're making smaller edits?
Some of these comments miss the advantage of diffusion. This is will have a big impact on edge devices, such as your phone or the GPU in your computer.
An LLM's decoder computes tokens one-at-a-time because attention has to account for each previous token. The existing LLM decoders scale well when you have enough load to batch many inferences together. Diffusion of limited benefit there. On edge you have a different problem: your inference accelerator is starved while sloshing GB of weights back and forth from RAM. That's because the consumer RAM like LPDDRx/GDDRx is lower bandwidth than HBM, and the requests are serial so you can't batch compute common weights. Diffusion can compute tokens in parallel which relieves the memory bandwidth bottle neck.
Edge devices don't just have limited memory bandwidth though, they also have very limited compute. To the extent where you don't actually need all that much batching to saturate their viable compute and run into obvious thermal/power limits. (It's just not true that "requests are inherently serial" in edge inference; any time you have multiple requests (i.e. "chats") in flight, batching becomes applicable if you have enough memory capacity for the KV caches.) I'm not sure how diffusion models are supposed to help there, if they simply take more compute for lower-quality outcomes and a dubious saving in memory bandwidth.
Forgot to mention it previously, but this might be a good model for a narrow slice of midrange systems that really are more skewed towards compute than memory bandwidth, but also don't have enough memory capacity to effectively use batching. (E.g. top-of-the-range consumer GPUs, or earlier generations of datacenter GPUs.) Although you do also compete with things like MTP there, which is targeting a similar tradeoff, or with denser models featuring a similar amount of total parameters. So I'd say that the jury is very much still out, even in that narrow space. Diffusion models are also apparently very hard to scale to a hundred-billion or trillion parameter count, since the way you train them is completely different to the usual one-token-at-a-time models.
You’re mostly right but conflating attention with autoregressive/causal which is the real issue that prevents you from using more compute
You can use diffusion with attention, and this model does in fact use attention
Yes, I should have said autoregressive.
Google keeps flexin'. It's surprising that Gemini isn't more competitive against Claude or OpenAI models for code and agentic use, because it's clear Google still has some of the best AI people in the business. But, I guess Google is focused on stuff that runs on phones and near-realtime use cases, rather than the big thinky LLMs.
All these efficiency improvements seem likely to be really important to the future of AI, though, as the money starts flowing the other direction. The days of subsidized tokens to try to lock people into specific ecosystems are coming to an end, and we're going to have to start paying what it actually costs.
The companies that figure out how to make it cost-effective to run really smart models are the ones that will win. DeepSeek costs an order of magnitude less than GPT 5.5 or Opus 4.8. It's worse than either, but not catastrophically worse. I'll happily pay ten times as much for the best coding model, because it saves enough human time to justify it, but not a hundred times as much, which is where things seem to be heading (GPT 5.5 Pro cost over 200 times as much as DeepSeek in some benchmarks I recently did, and ~30 times as much as Opus 4.8).
Google is clearly gimping the gemma models. There is a 122b gemma 4 that was never released, but was a part of the announcement tweet. Plus they weren't going to release MTP until people figured out they're running it on the pixels
I dunno about that. Gemma 4 is probably the best model for general self-hosted use for almost everyone that doesn't have a data center in their basement. They didn't have to release it at all, and they didn't have to release speculative decoding drafters, and they didn't have to release the QAT version of the models that makes the 4-bit quantization perform very close to the bigger versions, and can run in 32GB. I'd love a 122B version of it, and I didn't realize they'd ever announced one was coming (though I remember there being speculation about it). But, also, I'm happy they're doing so much with it. They've got all the sizes covered, it has great prose for an LLM, better prose than even most larger models, it's got great audio and vision, and broad language support. As self-hosted general purpose models go, it's the total package.
Qwen 3.6 is maybe better for code (though I'm beginning to think otherwise after some benchmarking I've been doing, where Gemma 4 has been overperforming expectations), but for just about anything else, Gemma 4 is the one.
If they're gimping it, why is nobody else making a better one that small?
Fable's costs are twice Opus' and it's clearly quite competitive with GPT-Pro, so that seems like it might be a good option for you if the trigger-happy safeguards aren't too much of a problem. Google has their own "Deep Research" option in this space which seems to work well.
The nice thing about DeepSeek is its ability to be run on local hardware, with no API costs involved. If you care deeply about that, then it being a bit worse than Opus or GPT isn't really a problem.
I think Google will win out in the end. They are concentrating on what matters, performance per watt, and performance per dollar. They are building their own inference hardware and are working towards edge-computing which removes latency and compute overheads. These big LLMs are not yet cost effective, Google is just letting them burn their investment funds to "sell" to consumers at below cost.
After the AI bubble bursts, it will be the likes of Google that come out the other side still wearing their shirts. I think this bubble is out to scalp some giants.
NVIDIA are hosting a free endpoint for this one, details at https://build.nvidia.com/google/diffusiongemma-26b-a4b-it - you have to create an account and (I think) verify a phone number too.
(I got it to draw a pelican: https://tools.simonwillison.net/markdown-svg-renderer#url=ht... )
Maybe with very fast models you could request animation frames, e.g., frame 1) right foot at 12, left foot at 6; frame 2) right foot at 3, left foot at 9, etc.?
And instead of reporting tps, you would - of course! - report pfps (pelican frames per second).
I register few weeks ago, the account still not verified, despite following the procedure. Can't use API if the account not verified.
Thought of you this afternoon, "after you click the record button can you make a 'boop, boop, boop, clack!' like a lead-in from a from a clapboard (using web audio synthesis apis)?"
...was quite surprising the result!
A good visual explanation of how text diffusion models like DiffusionGemma work: https://newsletter.maartengrootendorst.com/p/a-visual-guide-...
> DiffusionGemma reverses this inefficiency. Instead of predicting words sequentially, it drafts an entire 256-token paragraph simultaneously. By giving the computer's processor a larger chunk of work at once, DiffusionGemma utilizes your hardware to its full potential. It upgrades your model inference from a single, sequential typewriter to a massive printing press that stamps the entire block of text simultaneously.
> Operating as a 26B total Mixture of Experts (MoE) model that activates only 3.8B parameters during inference, DiffusionGemma fits comfortably within 18GB VRAM limits of high-end dedicated consumer GPUs when quantized.
Okay, so Gemma 4 26B is a MoE model that's really fast on my 24 GB GPU using ollama. This sounds like speculative decoding but I don't think that works with MoE models? It's hard to keep up with all this when it's not your job to keep up with it.
This is a different model with, confusingly, approximately the same number of params as the existing gemma4 MoE. Unclear from a quick scan whether one was trained somehow from the other.
The mechanism isn't the same as speculative decoding. Speculative decoding happens sequentially and (usually) a couple of tokens at a time; diffusion doesn't, and does blocks of text at once. I haven't read the collateral yet but my assumption would be that it's trained to keep the specific experts stable across a diffusion block.
Thanks. I found this other comment that links to a very thorough explanation: https://news.ycombinator.com/item?id=48479042
Oh, fascinating. So they did reuse the existing gemma4 MoE.
A few days ago I was just thinking that Google never talked about their diffusion text generation model after demoing it at I/O a year ago. The rumor is that it was too expensive to run, but with the provided chart using the same 1x H100 hardware and comparing DiffusionGemma to regular Gemma, that shouldn't be the case. I'm curious what the downside for this speed is here aside from being slightly weaker than Gemma.
> I'm curious what the downside for this speed is here
"DiffusionGemma's speedup is designed for local and low-concurrency inference. In high-QPS cloud serving, autoregressive models can be deployed to saturate compute efficiently, so DiffusionGemma's parallel decoding offers diminishing returns and can result in higher serving costs"
Well with a standard autoregressive model you can generate for example 256 tokens at once if you have 256 users, with this approach you can generate 256 tokens for a single user but you need several forward steps.
So the diffusion process takes more GFLOPs, if you have enough users you can already balance memory and compute.
Batching is a fair counterpoint.
[dead]
It is cool but local models while okay already feel noticeably worse than even the cheapest APIs so I can't see myself sacrificing even a little bit of their quality for speed. I'm sure it's worth it for some usecases, curious to hear specific ones that people are already planning to deploy to production.
Maybe writing / bootstraping unit tests?
Does not need opus level to write, and easy to iterate on.
I can see it but even if I do that for something like tests I'd still eat the time cost of the normal Gemma for 10% extra performance. And further, if you switch between the fast and normal Gemma for different tasks you eat the big time cost of loading the other model (and maintaining both in the first place).
What would a diffusing reasoning model look like? have a pre-defined length [thinking] block that gets diffused over a long time, and then the final output block uses what is in that thinking block as part of its input? And how do diffusion models decide the output length in the first place, is it a pre-set parameter? or does it diffuse an [end] token into the middle somewhere?
got one answer by reading the rest of the comments, makes sense that the diffusion process is inherently reasoning-like: https://www.inceptionlabs.ai/blog/introducing-mercury-2
I just *love* the commit message on Github: "Make TPUs go brr"
Is the diffusion approach any use in Multi-Token Prediction (MTP) drafters? https://blog.google/innovation-and-ai/technology/developers-...
Yes, DFlash is currently a SOTA speculative decoding method that Xiaomi just used in their MiMo model for >1000tkps
MTP is a training optimization. Drafting requires verification, and verification is the full model inference. Speculative decoders are the name for the inference time optimization, that is more like a verifier that is a smaller model.
Has anyone evaluated any diffusion LLMs for error spotting?
E.g. run your normal autoregressive LLMs (with MTP whatever, as you like), then run a single diffusion pass over the result, and observe any tokens that diffusion thinks are unlikely.
Then prompt the autoregressive llm with some structured reasoning "<think>Is <diffusion unlikely part> an error? .."
Because the diffusion model is so structurally different perhaps it makes different errors such that this would provide gains even vs running distinct autoregressive LLMs which often make the same errors.
The same argument could apply for RWKV but it would be relatively expensive to apply it as a second pass on a big block of output, while it seems like a diffusion model would be cheaper.
Thanks gemma team for this release.
Compared to autoregressive decoding, diffusion is huge for local MoE inference because of the improved token generation efficiency, especially for normal GPU + ram offload setting.
However, there are models which are better positioned on the performance vs memory pareto front, i.e. dense models, so I'll just wait.
P.S. QAT is really something as it reduces the performance fluctuations compared to the normal one. Thanks again.
I'm not getting anywhere near the speeds advertised on my 3090 Ti, alas, but it's fun watching it "fill out" its answers. I did Simon's "SVG pelican on a bicycle" test on it and the result was quite minimalistic but fit the brief: https://gist.github.com/peterc/7672e74ec1437945e5fca5ce2c1c9... -- this was on the Q4 quant running on patched llama.cpp. I will be interested to see if Simon's looks much different.
Hi! What implementation are you using? Right now VLLM is the one recommended. llama.cpp is in an early draft
Yeah, the patched llama.cpp. The reason is I saw that using the Q4 quant on vLLM is discouraged and the int8 won't fit on my 3090 Ti, but I could certainly give it a go. I also skipped Transformers as it needs to download the full weights and quantize them locally and I didn't fancy waiting for a 50GB download.
Maybe someone can explain: in image generation some models are already using rectified flow. Which was hailed as the next big thing. Are we going to see discrete rectified flow models next or is that unlikely?
Does anyone know of the current intrinsic limitations with Diffusion text models compared to autoregressive?
I ran this question by ChatGPT and Claude and they came up with limitations in GRPO RLVR, but I'm not sure..
The intrinsic limitation of text diffusion is that natural text contains serial dependencies where a word at the beginning of the text strongly influences what comes later, and if there is a long enough dependency chain within a diffusion block, the small number of diffusion steps may not be enough to resolve all dependencies, so that you end up with incoherent output.
The obvious solution is to simply do more steps for larger sequences though, right?
How exactly does this work with CoT?
CoT legibility largely disappears which is quite concerning from a safety perspective
I'm curious how diffusion models do at tool calling, curious what wins there are there.
The video demo of the svg sword is an interesting example of what is so interesting about diffusion models: it's not just putting one token after another to make edits to a file. It's skipping around, it's re-editing previous lines. I feel like forcing it to write too calls is maybe not its best nature.
I feel like perhaps instead of a monolithic edit file tool call, perhaps the diffusion model would be better suited to posting a change stream, a series of edit ops, across multiple files.
It's nice that Unsloth has already published the model on HF, but it requires a fork of llama.cpp to run at the moment.
Can LoRAs be used to increase the quality of these diffusion models? Nvidia mentions something about this https://huggingface.co/nvidia/Nemotron-Labs-Diffusion-8B#inf...
Yes, full ft or lora https://github.com/NVIDIA-NeMo/Automodel/blob/main/docs/guid...
I think this is the future. The sort of left-field rumble that turns into a quake in 5 years.
Almost certainly not if things remain as they are. The reason there's been little traction is the quality gap between diffusion and autoregressive models is pretty stark. I mean just look at the benchmarks here. Large dropoffs, with the hardest benchmarks seeing the largest drops. On top of that, almost all the speed benefits of diffusion models become negated at scale. So this is only attractive for local model development and almost everyone training local models still care about pound for pound quality and inference efficiency at scale.
It's fast enough that "ask it twice and pick the best" should still come out ahead performance-wise. I don't know how much that would close the quality gap by, but it's worth a play.
This may be the future of local models.
The thing is, diffusion models perform somewhat worse than autoregressive on text. So you lose some performance.
Speed is the big advantage. Autoregressive when doing local inference is mostly memory bound; you're doing one token at a time, for each token you need to load all weights. MTP helps a bit by allowing you to draft tokens in a smaller model and then verify them in parallel with the larger model, allowing you to do a few computations for every memory load, but because you're still doing tokens sequentially and need to discard invalid drafted tokens, you can only get so much speedup.
For hosted models, however, you can batch many token generations together, fully utilizing all of the compute while no longer being bottlenecked on memory bandwidth. So they are already operating at close to max efficiency.
So, diffusion kind of loses its beneifit in hosted models. Sure, maybe you could pay more to have slightly lower latency responses by doing diffusion for one user at a time instead of autoregressive for many in parallel. But given that it also reduces accuracy, it's hard to see where you'd really want that. Unless they're able to bring it up to par with autoregressive, it seems like it's a bit of a dead out outside of local models where you're generally just doing one thing at a time.
I'm particularly curious to know how this plays out, and I seriously hope that more labs focus on diffusion models for text usage.
My immediate thought - this performs slightly worse than the autoregressive gemma equivalent, but it may also let me functionally run better models in diffusion variants.
Ex - I can run 70b-120b autoregressive models locally right now, but I get ~5-15t/s, which just isn't fast enough for serious work.
Which caps me down in the 20-36b models (ex - gemma4) where I can get 100+t/s on the same hardware.
So the question becomes - does the quality drop from a diffusion model outweigh the quality bump from using a larger model?
Because if not... sounds like diffusion models have a lot of space to thrive.
---
Sadly - if they can't be hosted profitably, I question whether this space will actually be explored.
This is a place where I could see Apple actually investing serious money in AI research to sell devices.
Is anyone doing text diffusion in latent space instead of tokens?
Do diffusion models support tool calls? If so is the tool call support on par with autoregressive models or worse? (edited spelling)
Any text generation model can easily be made to support tool calls.
omlx.server - WARNING - POST /v1/chat/completions -> 400: Tool calling is not supported with diffusion models.
it just me that thinks its kinda weird that they conflate speed in tokens/second and latency, when i think of latency as time to first token? like it generates an entire paragraph of tokens faster but wouldnt it still be slower if your reply is only 1 word because it has to do the entire 256 tokens as a chunk
I always thought that fundamentally diffusors were the cooler idea of the two
I wish labs would do QAT and release these quants, at this point looking at releases of bf16 without QAT feels like looking at half backed bread, we can quantized it but it is not the same as QAT. Or I am missing something here ?
[flagged]
I can’t help but feel like there’s something here that will matter for future LLMs.
The bidirectionality could be a big deal: being able to refine a sentence with both left and right context feels closer to how editing/thinking actually works than committing to each token forever.
Maybe the current models aren’t good enough yet, but the direction feels important.
[dead]
[dead]
We need more local open weight models that are performant and just as good (or good enough) as the best frontier ones.
Then you will be able to achieve Jevons Paradox and enjoy the same “productivity gains” without paying for these extortionate token prices by closed model providers or have it as cheap as possible.
And especially, no silent nerfing of the model.
We have this though, right? Compare SOTA local models to where the frontier was last year. There weren't many people complaining that last year's frontier models were incapable.
Next year, and the year after, Fable, GPT 5.5 and Gemini 3.5 will feel quite ordinary. And perhaps even within reach of a prosumer running models locally.