not op but signal has basically solved this. users are not just randomly accessible by anyone. they can share a long URL that contains an ID, or make a short username they like and pass around to people. and even then the recipient has to accept being contacted by each other user

true that signal now relies on the phone number system for trust and safety, but that's not core to how signal works, it could be replaced if they really wanted.