The way to solve the problem you have is to maintain a receptacle specifically for untrusted interactions. You might try to minimize them but they will eventually become unavoidable like now.

Back in the day before email it was called a blind box and you’d rent it at the post office.

In your situation you’d just keep a second phone/number.

It costs more money, there isn’t a free alternative or do it yourself method.