At risk of getting more technical, some near-future combination of bitnet-like ternary models, less-autoregressive architectures, taking advantage of sparsity, and models not being so stupidly general-purpose will bring inference costs down dramatically. Like, a watt or two on your phone dramatically. AI energy cost is a meme perpetuated by Altman so people will give him money, kinda like a NFT scheme.

...In other words, it's really not that big a deal. Like, a drop in the bucket compared to global metal production or something.

The cost of training a model in the first place is more complex (and really wasteful at some 'money is no object' outfits like OpenAI or X), but also potentially very cheap. As examples, Deepseek and Flux were trained with very little electricity. So was Cerebras's example model.

I happened across a podcast episode that was about AI, that I was listening to with friends. I don't know if you want to take away anything from it but I figured I'd mention it here in case anyone wants to. Look for Serious Inquiries Only episode 477, "Debunking Bad AI Research, and Bad Coverage of AI Research". For you it might not be super interesting, since it's trying to explain the matter to those who might not already know much, debunking some bad studies, but towards the end they talk about the environmental impact. And this is with two experts, I believe.

One thing that pops up there is that training a "moderately large" model requires produces twice the CO₂ output of an average American over their entire lifetime. They mention water usage is really bad, too. And "moderately large" refers to what a University research team might be cooking up. Big companies have magnitudes more environmental impact from training their huge models.

(There is also a part 2, with the followup episode.)