It's their flight pattern for me too.
Also, I have never been in a place with enough butterflies that some would get in my face, or I might have ended up smacking some that catch me distracted.

That's interesting, there's something to that. They are hard to predict, and it puts me on edge. When they're just perched somewhere, it bothers me less.