I've had similar trouble understanding this aspect in the past, so I'll share how it now makes sense to me, within the frameworks of existing IP law. IANAL.

Copyright law protects a specific, tangible work from duplication, meaning that if someone directly derives an unauthorized copy from an original work (eg scanning a book), then that's copyright infringement. However, copyright law does not simply look at whether a copy is identical to the original. There must be an analysis of the provenance of the copy: did the copy come from the original?

Two people can separately stand at the Grand Canyon visitor center and use their own cameras to take identical film photos from the same vantage point, such that the developed film is identical in every way. And yet, neither has violated each other's copyright, because neither photo came from the other. Person A can sell their copy to a magazine, while Person B distributes copies for free with their holiday cards. This is all A-OK.

The trouble is when somebody suspects infringement, because they see a copy that looks unusual similar to an original. If they decide to raise a copyright lawsuit, then the defense would like a way to affirmatively show that "no, our copy didn't come from your original". How do they do this?

In software, the approach would be to methods like the clean room approach. This is explicitly a method for legal reverse engineering, precisely to avoid copyright concerns. It works because while copyright protects a tangible work, it does not protect the ideas recorded by that work. So if somebody was given only the idea, and tasked with writing the software that re-implements that idea but without ever seeing the original, then the resulting work cannot possibly have been based on the original work. Rarher, it's just a take on the same idea, which is allowed to be similar, in the same way as the photo of the Grand Canyon. There is no straight line that connects the supposed copy to the original work, apart from the idea itself, which copyright does not cover.

With LLMs, there is zero assurance that any clean room approach was followed. By training directly on original, copyrighted works, there is a cloud of uncertainty about whether the LLM is regurgitating parts of the original (which would be infringement) or if it actually reduced the user's prompt into an idea, and then independently generated code which implemented that idea.

Copyright infringement poisons everything that is derived thereafter, so even the spectre of a doubt causes problems, because if GNU or whoever else builds atop questionable copyrighted software, that foundation collapse suddenly if someone brings a lawsuit to enforce their copyright. And those are pricey to defend, even if it's without merit.

The clean room method, when followed faithfully, is a way to discourage potential litigants from even filing a lawsuit, because it's clearly obvious that they would fail to prove a connection to the original work. Patents are a different story (where ideas are protected).

LLMs basically have the same problem as when an engineer quits and goes to work for a competitor. Will they have (accidentally or not) taken trade secrets with them and implemented those in the competitor's product? A well-advised company would go through great lengths to make sure that engineer isn't working directly on a competing product, to avoid a costly lawsuit. LLMs do no such mitigation.

I think the fundamental difference is that a human trained on something isn't likely to reproduce the thing they were trained on exactly.

I know that some research I read a few years ago showed that LLMs generally produced novel text (rather than simply returning verbatim training material), but that that wasn't always true, especially as the training sample got smaller, or a single training text got larger (something like a 1000 words).

So sure, for something simple like creating super standard CRUD or basic API endpoints in common languages, the sample size is significant enough that what you get is likely to be homogenized from thousands of samples available, and not likely to be an exact copy of someone else's code. But the risk is that as the problem being solved gets more niche, and the training sample size gets smaller, you become more likely to get exact copies of someone else's code, or something close enough that it might be considered plagiarism.

There's also still grey areas where some courts in some areas have stated that LLM generated content can't be copyrighted, since a human didn't create it, creating issues with protecting sources with copyright.

We humans are not Ai. Don't mix that up. In example just because you can write a script to scrape data from all Github, does not mean you can use the scraped data as you like without giving credit and respecting the license. But that is what Ai programs do. LLMs are not human and have not the same human rights.

I don't understand specifically the copyright issue with respect to "trained on x data"

It's a question of whether the output of the LLM is a derivative work of the training data. From a copyright standpoint, if the work is derivative (and not fair use), then it needs to follow the terms set by any licenses you have to its parent works.