To partake in society you have to accept societal contracts. One such contract is to not be a dick to others. If you don't vaccinate yourself against certain things, you are liable for spreading the disease. And thus you are being a dick. And thus you break the contract.
If you excuse yourself from society going forward though, I see no problem with your stance.