This is like trying to argue that JK Rowling is fine because there are people that have never read Harry Potter and yet still turned out bad.
You're right that the bible is used as an excuse for people to do what they wanted to do anyway, but that's true of practically any bigoted or sexist text. There's no shortage of men who wish they could tell their wives to shut up, but that doesn't mean its a good thing that there are books that justify and encourage them.