Hmm… I think of difficulty, or lack thereof, as integral part of the fun. I think they're inseparable, essentially.

I don't really enjoy the process of learning and getting better at 3rd person shooters, for example, so I don't typically enjoy playing those on higher difficulties. If I pick one up, I know I'll most likely have more fun playing on lower difficulties because it eliminates a process I don't really enjoy. In other words, shooting shit is still fun, but I need the difficulty toned all the way down to enjoy it.

Conversely, I love learning the intricacies of combo systems of action games and figuring out how to exploit enemies and whatnot, so I have to play those on the highest difficulty to get the full experience and have the most fun.