In Tox you have a code on the end of the Tox address. One can do similar, but have different codes for different levels of acceptance. Default - ignore. Some other code - add to the list of callers without notification. Some other - with notification. Some other - for SMS, but not calls, or the other way around. And so on.

The problem with things being memorable exists, yes. Computers can make calls, meaning that there's no solution. A good secret required to call someone can't be memorable.