A few weeks ago, Gemini got confused when it tried to go first as black multiple times, so that's the most immediate one I can remember. Last week, chatGPT offered to set up chess puzzles for me, but it made mistakes 3 out of 3 times.

Maybe I'll try again. Is there a certain one you've seen good performance out of?