Everything they can do, a go-pro or adjacent camera would work without issue and with less ambiguity.

So why aren't we banning those for perverts too?

Lack of imagination. One can easily imagine scenarios where glases can be useful outside filming other people. Simple tasks like getting a receipt and scanning it before throwing away, or getting dietary feedback on a menu items would be quite more easily achieved with smartglasses than otherwise. AR-assisted navigation in cities with to image data could be quite useful as well.

Not to mention if one has sight issues; I can imagine describing environments or reading texts being wildly useful for some.

Perhaps consider first banning devices that are designed for secret filming, e.g. "car key" cameras, watch cameras, pen cameras, powerback cameras, and yes, glass cameras... You can find them from Amazon for <100€, which is much more affordable option in the first place.