They are statistical prediction machines. The more they output, the larger the portion of their "context window" (statistical prior) becomes the very output they generated. It's a fundamental property of the current LLM design that the snake will eventually eat enough of it's tail to puke garbage code.