Am I crazy for thinking that?

I don't think so. It's one of the reasons I go overboard in my server security and monitor my pfsense logs constantly.

Not at all. Anything an attacker can connect to is a potential vector.

Keep in mind, this goes for outbound connections as well. If something is compromised server-side, and you or your software reach out to it, that's also a potential vector. Less likely, since the attacker has to compromise that server and pass their attack off as legit traffic, but it's possible.