Even if the Sohu Asic chip does attention part super fast, wont the bottlenecks come from, when this data goes to some next stage ?
I'm just thinking whats the chance that we will actually start using ASIC chips for certain parts of AI inference.
Taalas uses a design that requires masked ROM layers specific to a model, but Etched uses a newer design with a specialized RAM that supports any transformer model.
2 hours and no comments makes me wonder if the Etched chip exists or not.
> Sohu figures are per chip, derived from Etched's published 8-chip server claim of 500,000 tok/s on Llama 70B; not independently verified. (claimed by Etched)
So these are not verified benchmarks and they are all claims. Again is this chip real?
>That per-chip advantage is real in the sense that it reflects Sohu's architecture: the chip hard-codes transformer attention directly into silicon as fixed-function logic rather than as software running on a programmable compute unit.
Is anyone else disappointed by the fact that they don't have any real edge? At least not on the basis of what they advertise as their special sauce?
Attention is compute intensive during prefill and memory bound during token generation.
The reason is quite simple and boring. Attention is two matrix multiplications interrupted by a softmax. Flash attention is just the online version of attention by making clever use of an online softmax function so that the kernel can be fused.
So yeah they can accelerate flash attention even further by putting it into the hardware, but the truth here is that you still spend most of your time in the matrix multiplications. This is not a place where you can get more than a 2x edge and even that is optimistic.
You simply cannot get around the fact that token generation requires you to read the entire KV cache once per token and that requires memory bandwidth more than it requires a flash attention accelerator.
And then there is also the boring reality that the feed forward network (FFN/MLP) matrices are often chosen to have a size so that at short context sizes, the FFN ends up having a fixed cost that dominates so you are back to optimizing matrix multiplication. It's only once the context is getting really long that the quadratic cost of attention starts dominating everything.
Of course for video and image models, the input image is turned into patches and that means the context window is always pretty big even for minimal inputs, so that's an area where accelerating flash attention is pretty important.
>Etched Sohu is a transformer-only ASIC, and Etched AI claims one 8-chip Sohu server delivers 500,000 tokens per second on Llama 70B, roughly 62,500 tokens/sec per chip.
Ok, but that's not possible without also scaling the memory bandwidth and that sounds like the actual innovation behind the company.
>There is no way to compile a convolution, an SSM scan, or a diffusion U-Net step onto Sohu because there are no programmable units to target.
im2col is a pretty simple way of mapping convolutions to matrix multiplication. There is no reason to abandon support.
>Sohu uses 144GB of HBM3E per chip with approximately 1.8x the memory bandwidth of an H100 SXM5.
Ok, so they have no edge on memory?
>What Sohu Cannot Do
>Vision and multimodal models: any model with a visual encoder (LLaVA, Qwen-VL, LLama 3.2 Vision) cannot run on Sohu because the encoder uses convolutional or attention operations outside the pure transformer pattern
Ok now we reached a point of absurdity. If you have a transformer ASIC implementing Flash Attention directly, your biggest customers would be people who spend more energy on the compute bound parts of attention, such as vision models that take images or videos as input. The flash attention part of the ASIC would be getting exercised all the time. Worst of all, there is no clarification whether this affects ViT models or not.
>The conditions are important: this appears to be measured at or near batch size 1, where the fixed-function attention circuits can operate at peak efficiency without the batching overhead that GPUs exploit to amortize their lower per-token bandwidth.
Ok fine, at least they have one place where they have the upper hand, but I still don't understand the batch size 1 argument since batching the FFN/MLPs objectively decreases the memory bandwidth you need to process multiple sessions.
Even if the Sohu Asic chip does attention part super fast, wont the bottlenecks come from, when this data goes to some next stage ? I'm just thinking whats the chance that we will actually start using ASIC chips for certain parts of AI inference.
Can’t wait for single chip asic qwen3.8 27b
It’s small so should be cheap per chip. And it so much smarter than it ought to be for its size.
Problem is asic take forever to cut and are always months behind the latest open weight sota
Taalas uses a design that requires masked ROM layers specific to a model, but Etched uses a newer design with a specialized RAM that supports any transformer model.
2 hours and no comments makes me wonder if the Etched chip exists or not.
> Sohu figures are per chip, derived from Etched's published 8-chip server claim of 500,000 tok/s on Llama 70B; not independently verified. (claimed by Etched)
So these are not verified benchmarks and they are all claims. Again is this chip real?
Keep on eye on TechTechPotato (https://www.youtube.com/channel/UC1r0DG-KEPyqOeW6o79PByw) for updates. Ian has put enough pressure on other companies to get them to allow third-party verification.
>That per-chip advantage is real in the sense that it reflects Sohu's architecture: the chip hard-codes transformer attention directly into silicon as fixed-function logic rather than as software running on a programmable compute unit.
Is anyone else disappointed by the fact that they don't have any real edge? At least not on the basis of what they advertise as their special sauce?
Attention is compute intensive during prefill and memory bound during token generation.
The reason is quite simple and boring. Attention is two matrix multiplications interrupted by a softmax. Flash attention is just the online version of attention by making clever use of an online softmax function so that the kernel can be fused.
So yeah they can accelerate flash attention even further by putting it into the hardware, but the truth here is that you still spend most of your time in the matrix multiplications. This is not a place where you can get more than a 2x edge and even that is optimistic.
You simply cannot get around the fact that token generation requires you to read the entire KV cache once per token and that requires memory bandwidth more than it requires a flash attention accelerator.
And then there is also the boring reality that the feed forward network (FFN/MLP) matrices are often chosen to have a size so that at short context sizes, the FFN ends up having a fixed cost that dominates so you are back to optimizing matrix multiplication. It's only once the context is getting really long that the quadratic cost of attention starts dominating everything.
Of course for video and image models, the input image is turned into patches and that means the context window is always pretty big even for minimal inputs, so that's an area where accelerating flash attention is pretty important.
>Etched Sohu is a transformer-only ASIC, and Etched AI claims one 8-chip Sohu server delivers 500,000 tokens per second on Llama 70B, roughly 62,500 tokens/sec per chip.
Ok, but that's not possible without also scaling the memory bandwidth and that sounds like the actual innovation behind the company.
>There is no way to compile a convolution, an SSM scan, or a diffusion U-Net step onto Sohu because there are no programmable units to target.
im2col is a pretty simple way of mapping convolutions to matrix multiplication. There is no reason to abandon support.
>Sohu uses 144GB of HBM3E per chip with approximately 1.8x the memory bandwidth of an H100 SXM5.
Ok, so they have no edge on memory?
>What Sohu Cannot Do
>Vision and multimodal models: any model with a visual encoder (LLaVA, Qwen-VL, LLama 3.2 Vision) cannot run on Sohu because the encoder uses convolutional or attention operations outside the pure transformer pattern
Ok now we reached a point of absurdity. If you have a transformer ASIC implementing Flash Attention directly, your biggest customers would be people who spend more energy on the compute bound parts of attention, such as vision models that take images or videos as input. The flash attention part of the ASIC would be getting exercised all the time. Worst of all, there is no clarification whether this affects ViT models or not.
>The conditions are important: this appears to be measured at or near batch size 1, where the fixed-function attention circuits can operate at peak efficiency without the batching overhead that GPUs exploit to amortize their lower per-token bandwidth.
Ok fine, at least they have one place where they have the upper hand, but I still don't understand the batch size 1 argument since batching the FFN/MLPs objectively decreases the memory bandwidth you need to process multiple sessions.