Smartphones haven't been called "pervert boxes". It's the implication of what someone that would buy and wear the glasses would also be able to do.
We implicitly trust doctors just because doctors are generally trustworthy for body things. At the same time, the pervert glasses are called pervert glasses because people have used them for that stuff and we implicitly distrust people that wear them because of the implication.
So if I see a doctor with pervert glasses on even go they are not recording, I'm immediately distrustful of their phone too. The trust of them being a doctor is immediately broken.
I hope I was clear, it's not if the phones can more easily record or not, it's the current general perception of those glasses and how it warps peoples perception of others.