You use them minimally and responsibly, you don't over-rely on them. Is there a problem in that case?
Responsibly is the key word there. I.e. vibe-coding an entire project full of security risks without understanding the architecture or languages used. Vs. using LLMs to error check code, brainstorm a project, write code you feel confident you can review yourself.
Also, like you say, hosting locally. Or using a provider like Mistral that is European, not evil, uses France's abundant nuclear power...