Apart from just bad gamedesign which is subjetctive in itself i'd say it mainly boils down to genre preferences for me. I have my two and half genres which i really like and i'm really good at. With them its almost always relaxing.
But tbh the neat part with gaming is you can just walk away. Like sometimes i feel like to put in the work to overcome a challenge. eg i just finished NFS Most Wanted 05 last week for the first time even though i fucking hate the excessive rubberbanding and the police chases(controversial i know but thats my personal take). But the struggling was worth it in the end because i fucking love racing games and finally beating one of the most loved racing games of all time after 20 years was neat.
But there are so many games i just dropped because they annoyed me in some way or another. I seek fun. Either embracing the challenge is fun or its not. I'd rather seek a new challenge then embracing chores in my free time.