They are like stars in the sense of orbital mechanics.

But a star can be completely understood by the laws of physics we know. While a black hole breaks our understanding and we have no idea what's going on in there.

It's the fear of the unknown.