That paper never passed peer review, and I believe it was trying to find the total footprint. Training probably does take a lot (considering it involves building and running a data center for 3 or 4 months), but once a model is trained you can run it locally on a regular mid level PC.

It's unlikely running models locally on your own PC uses a glass of water per question. But there is a large environmental cost to building and running so many data centers or asking the latest LLMs running that way.

A lot of that environmental cost comes from the fact Governments have asked data centers to not interrupt the power grid, and billionaires have subsequently responded by choosing the dirtiest ways to power them so they're off grid.

https://duckduckgo.com/?q=dirty+power+ai+data+centers+powered+by

This was a choice made by governments, billionaires, and their investors.

But yeah, if a model was trained in an environmentally powered data center then run locally it wouldn't be such a problem.