I think there are many other significant inference improvements which an be built out that are being ignored because the API between client and inference stack would be tricky to nail down.
This is not a criticism of the research but instead the presentation but the repo looks very fishy, it isn't clear that this is from a bunch of researchers from Berkley. It also does not make very clear (on the GitHub) what optimizations or performance they are targeting.
Looks to me like it allows you to resources you wouldn't normally be able to use. CPU, RAM, GPU, VRAM, etc in order to run models that you normally would not be able to due to the models being larger than the amount of vram on the gpu.
I think that's the idea. It looks weird so I didn't spend much time really looking.
The linked paper is a much more interesting read: https://arxiv.org/pdf/2608.16157
I think there are many other significant inference improvements which an be built out that are being ignored because the API between client and inference stack would be tricky to nail down.
This is not a criticism of the research but instead the presentation but the repo looks very fishy, it isn't clear that this is from a bunch of researchers from Berkley. It also does not make very clear (on the GitHub) what optimizations or performance they are targeting.
I have read the repo and website. I have no idea what this project does and what exactly its features are.
Looks to me like it allows you to resources you wouldn't normally be able to use. CPU, RAM, GPU, VRAM, etc in order to run models that you normally would not be able to due to the models being larger than the amount of vram on the gpu.
I think that's the idea. It looks weird so I didn't spend much time really looking.
[dead]