Why do you care so much about people you think are dicks thinking you're a dick? Just don't go. If you care about their opinion, talk with them. Otherwise it seems you're not overly fond of them, and they'd probably not miss you.