I recently come across this project, which seems very interesting because it let you run literally huge models on consumer hardware.

I am fancying yo try run Qwen3.8 flash next with this on my setup. I don't expect anything usable, but just for the fun of it. What stopping me is the download size of the model.

Has anybody tried colibri? Just to have an idea of speed to expect. 1t/s? 0.1t/s?

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 1 month ago*

They have some numbers in their Readme file. Seems to me for the model they tested, it's number of beefy graphics cards = Tokens/s...
Their rig with 6x RTX 5090 does about 6T/s, their laptop with one RTX5070 does 1T/s. And their computer without a GPU does about 0 T/s.

  • source