They can, and likely will, put a lien on your house which makes it harder to sell. Probably other avenues to fuck up your life too. Around me it's really hard to find a house without an HOA. It's probably like 90% of the market, or I triple my commute.
It's legally binding with the purchase of the house so there's no going around it.