My workplace has finally succumbed to the whole "we need to use AI to stay competitive" bullshit, and while my boss and my team aren't happy about it, he made it clear that execs would be looking for teams to have stories about how it's helped them work faster.
So, into the slop mines I went. Microsoft Copilot is the slop generator approved by our legal and infosec teams. I figured that I'd give it something easy, just pull me info on Exchange Online, Microsoft's corporate email system in Azure. Most of this stuff I'm already familiar with, so I'm just trying to see if it saves any time looking through Microsoft's documentation websites for specific details that they don't make obvious so you have to combine info from multiple pages.
I shit you not, every single response has had an innacuracy/issue. Many of them required follow up prompts because it gave me shit for Exchange Server instead of Online, despite how I set up the equivalent of the system prompt. I'm asking some pretty technical stuff, but it should absolutely be there in the training data.
Having a way to search the internet using natural language is useful, but that's not really what these things are. Natural language parsing is an amazing tool. I would love to see it stand on its own.
But the slop generation for responses falls flat, because it's trying to be everything for everyone. It doesn't know anything, so it can't reliably discern the difference between Exchange Online and Exchange Server (two different, but interoperable, systems with critical differences in functionality).
And all of that ignores that it's all built off widespread content theft, and the obscene resource usage, and that despite just how much shit is burning to make it work, the end result just isn't impressive.
TL;DR- It's not a threat to technology related "knowledge workers" either.