The duration really makes the difference here. I went to school in a tshirt at - 5°C several times because I wasnt outside for mor than five minutes in one go. But I really felt it when my train got cancelled one time and I had to wait 45 minutes in the shade for the next one.

Wind and humidity also play a big role. I am fairly cold resistant but as soon as you add wind to the equation thin single linen or cotton layers become inadequate for the occasion.