Yeah Idk. While I do in fact always have issues prompting AI... I kinda also think it has a weird form of "intelligence". I wouldn't say it does really well at following instructions. It tries. Certainly. But sometimes it rather does silly things. Like I recently was trying to prototype some web app. The app didn't work. So I told it to write tests. And it wrote some stub which always passed. I had an argument with the agent how it can't just tell me the project works now, but it always has to follow the usual agentic pattern: Plan, write code, execute the test suite, fix mistakes, repeat and only then add all files to a commit and tell me it's done. And now we need some tests for it itself to see whether the program runs. It just wrote super convoluted tests, launching some headless browsers, didn't ever get it right so ALL the tests kept failing. I told it to test functions and API endpoints instead (which is kind of a no brainer to do anyways). Those also kept failing. And after several hours of this nonsense it told me: TOO HARD, NEED TO FOCUS! And staged a commit which just deleted the entire test suite 😆 Hilarious. But neither what I wanted, nor what I instructed it to do...
I forgot what model that was. I was testing the Hermes Agent at that time.
I never know what amount of hand-holding is expected from me as the human operator. Models keep getting better. A year ago I had them make up convenient, yet nonexistent functions. Like mathproblem.solve() That hasn't hapoened in a while. But I can't be expected to instruct and teach it every tiny detail?!