Your thoughts are very similar to mine. The usefulness of machine learning to bridge the gap between the endlessly messy real world and the strictly regimented digital one can't be overstated, but throwing all your problems into an LLM chatbot is never going to yield good results. Case in point is the AlphaFold project which used AI to basically solve one of the hardest problems in modern biochemistry. They didn't do it by asking ChatGPT to "please fold this protein for me" they did it by assembling a multi-disciplinary team with a deep technical understanding of the problem and building a fully custom machine learning system which was designed from the ground up for the sole purpose of predicting protein structures. That's applied AI. All these so-called AI startups who's product is basically a coat of paint and a custom system prompt on the ChatGPT API are nothing more than app developers chasing the latest trend.
Incidentally, I had a very similar experience to your "Exchange Server / Exchange Online" problem. I was asked to make updates to an old VBS code base, and VBS is a language that I have neither experience with nor interest in learning, so I was using a chatbot to try and save a few minutes on some super simple beginner level questions. It kept confidently spitting out answers for VBA code, which is similar to, but very much not the same as VBS.