Google's built in AI for searches is hilariously inept about a third of the time.

That's when you notice it being hilariously inept. It actually worse than that, but in areas you're not expert enough to notice.

If they ever get around to changing fundamentally how it works, I'll give it another try. I don't like having to second guess everything it says, defeats the entire purpose of trying to use it If I have to verify everything. We made a similar but pared down output generator in college once, that worked in essentially the same way. We fed it a bunch of Freida Khala poems so all it would output was strange fragments of those. But that experience taught me, perhaps too well, how these things work under the hood. I can't imagine how many guardrails and parameters they have set up just to get it functional.

Sometimes it gives you sources that don’t have anything close to what it attributes to them. So you have to always be careful and double check.

It also still likes to make things up when it does not know the answer. Weasel words like „usually“ are a good indicator of that, although sometimes it uses them despite having solid sources and a definitive answer.