Hacker reverse engineers his Samsung TV and finds on-device AI listening for your dog and baby, watching what you watch, and borrowing your soundbar's mic if the TV doesn't have one. Then OpenAI banned him for reverse-engineering it.
Source:
It can never compete with the amount of computer power they have behind.
I recomend anyone with a decent enough computer to try local llms, it will give you a perspective on just the amount of horse power they have to run their models.
I have a 32GB machine with 9070xt with 16GB, of that what matters most (or at all) is the vram, which they're printing us out or out right denying selling to us. With that I can try running some more ambitious models of 30b parameters MoE.
The result in the best case scenario is a slow token output and not very credible responses.You also get frequently into reasoning loops, and tool using loops, and then run out of context.
I'm sure there's someways to optimize it that someone more knowledgeable can do, but to me served as a reality check of the discrepancy.
btw, my go to check prompt for models is "how many stars are there in Mario 64", and most models find it very hard to answer this, even when searching the web.