The answer is that it's messy and that I'm not qualified to say where the line is (nor, I think, is anyone yet). The generated parts are not copyrightable, but you can still have a valid copyright by bringing together things that aren't individually copyrightable. For example, if I make a manga where Snow White fights Steamboat Willie, I've taken two public domain elements and used them to create a copyrightable work.

So it's not like the usage of AI inherently makes a project uncopyrightable unless the entire thing or most of it was just spat out of a machine. Where's the line on this? Nobody (definitely not me, but probably nobody) really knows.

As for courts ever finding out, how this affects trade secret policy... Dunno? I'm sure a Microsoft employee couldn't release it publicly, because as you said, it'd probably violate an NDA. If there were some civil case, the source may come out during discovery and could maybe be analysed programmatically or by an expert. You would probably subpoena the employee(s) who wrote the software and ask them to testify. This is just spitballing, though, over something that's probably inconsequential, because the end product is prooooobably still copyrightable.

This kind of reminds me of the blurry line we have in FOSS, where everyone retains the copyright to their individual work. But if push comes to shove, how much does there need to be for it to be copyrightable? Where does it stop being a boilerplate for loop and start being creative expression?

I clarified this a bit in a follow-up comment, but my first comment was simplifying for the sake of countering:

[it's not in the public domain] because the actual human work that went into creating it was done by the owner of the AI Model and whatever they trained on.

Their claim that the copyright for AI-generated works belongs to the model creator and the authors of the training material – and is never in the public domain – is patent, easily disprovable nonsense.

Yes, I understand it's more nuanced than what I said. No, it's not nuanced in their favor. No, I'm not diving into that with a pathological liar (see their other comments) when it's immaterial to my rebuttal of their bullshit claim. I guess you just didn't read the claim I was addressing?