Without unblocking you'll be unable to message them. But if this isn't enough of a preventative, then removing them from your contacts and then turning off the "Allow From Anyone" privacy setting should do the trick. This would mean they still can't message you and you won't have them as a Signal contact. Aside from that, it's pretty much on you :/