Same, for me it's about perspective and what I'm controlling. In an FPS, I'm controlling the view window and I want it to directly follow my intention: non-inverted.
In a 3rd person game like Dark Souls, I'm controlling an external camera. It's the angle I'm viewing my character from and not necessarily the frame of the scene I am considering: inverted.
Flying games? Inverted. Rail shooter like Panzer Dragoon, I'm controlling the absolute position of a targeting reticle: non-inverted.
Those are just by preferences and how I perceive things, some people may take the opposite stand than me. But even those aren't strict rules, if a specific game doesn't feel right I'll try mixing it up to see if something works better for whatever reason.