I've been refusing to give them permission for about 5 years now, and the only ill effect I've really had is not having a name associated with the contact when texting them