98
A 25-Person Startup Built a Chip That Only Runs One AI Model. It's 73 Times Faster Than Nvidia. (dev.to)
submitted 8 months ago by to c/technology@beehaw.org
43 comments fedilink hide all child comments
 

Taalas HC1: 17,000 tokens/sec on Llama 3.1 8B vs Nvidia H200's 233 tokens/sec. 73x faster at one-tenth the power. Each chip runs ONE model, hardwired into the transistors.

you are viewing a single comment's thread
view the rest of the comments
[–] 6 points 8 months ago (4 children)

fpgas can sort of be a middle ground, but i don't know if they're capable of running llms

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 2 points 8 months ago (2 children)

    Is there such a thing as modular fpga so that you could "plug in" another one and add more gates, sort of daisy chain them? I don't know if such interfaces exist , sounds like it might need lots of bandwidth.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •