I hike year-round and camp March through October. In my experience, the more time you spend outside, the less sensitive you are to temperature swings. I do live in the PNW though, so I'm spoiled by the climate.

Having said that, I'm extremely sensitive to pretty much everything, and that probably also applies to temperature. I would probably relate to this meme if I didn't spend so much time outside.