I think that comes down to the genre and game. I've definitely played games where I was enjoying the story and wanted to see its conclusion, but couldn't be bothered with a boss rush in the middle of the game. In a similar vein, games with sudden difficulty spikes in the mid- to endgame portion might benefit from it.

At the end of the day, I'm a working adult, trying to fit in having some fun with all the other crap I need to do. I don't have time for games that need me to treat them as a second job to get good enough to make any progress in them, but games with random difficulty spikes or boss rushes that just serve to pad out play time by making you grind for levels or the ideal equipment or skills/summons out of nowhere feel like an annoying bait and switch to me.

Hmm… I think of difficulty, or lack thereof, as integral part of the fun. I think they're inseparable, essentially.

I don't really enjoy the process of learning and getting better at 3rd person shooters, for example, so I don't typically enjoy playing those on higher difficulties. If I pick one up, I know I'll most likely have more fun playing on lower difficulties because it eliminates a process I don't really enjoy. In other words, shooting shit is still fun, but I need the difficulty toned all the way down to enjoy it.

Conversely, I love learning the intricacies of combo systems of action games and figuring out how to exploit enemies and whatnot, so I have to play those on the highest difficulty to get the full experience and have the most fun.

I like to set the difficulty high enough to where I don't feel the game is too easy, but low enough so that I'm not getting frustrated and feeling like the challenge is bullshit.