AI can't write programs, at least not complex programs. The programs / functions it can write well are the ones that are the ones that are very well represented in the training data -- i.e. ultra simple functions or programs that have been written and re-written millions of times. What it can't do is anything truly innovative. In addition, it can't follow directions, it has no understanding of what it's doing, it doesn't understand the problem, it doesn't understand its solution to the problem.
The only thing LLMs are able to do is create a believable simulation of what the solution to the problem might look like. Sometimes, if you're lucky, the simulation is realistic enough that the output actually works as a function or program. But, the more complex the problem, or the more distant from the training data, the less it's able to simulate something realistic.
So, rather than building a ladder where the rungs turn into propellers, it's building one where the higher the ladder gets, the less the rungs actually look like rungs.