Even if you can't remove the HOA, they presumably have elected positions, so the people making them awful can be removed, even if that means standing yourself.