I use AI a lot and I've spent months trying to craft an instructional prompt that would produce even remotely human-like answers and it reliably fails at this every single time. I guess I can imagine how it could fool someone for a while who has never interacted with an LLM before but I honestly don't have a theory of mind for the individuals who are convinced about it. I've just grown so sensitive to the patterns of how it responds that I'm immediately disillusioned by it.

I sure hope we get there one day but we're not there yet and I honestly haven't seen a huge amount of improvement over the past few years. I still value it for the ultra-autistic thinking partner that it is but at no point does it give me reason to think there's much of anything "human" going on the other side.

It's unsafe.

When it acts like a human, I think we subconsciously think it'll decide things like a mentally normative human.

That's a very dangerous assumption, for sure.