LLMs are not thinking machines. They are word association probability engines. Most mentions of nukes would be in fiction...and the only reason to mention a nuke in fiction is for the nuke to be used or threatened to be used.

So, yeah. Maybe let's not rely on glorified autocorrect to handle our nuclear arsenals...

I think there was a movie on this hehe