Toilet fan rattles against sudden negative pressure outside

Oh man, I used to live in a very windy area and the bathroom exhaust vent cover would constantly be clanking. I forgot about that little bit of hell from living there.