The first couple times I saw the dress I saw it as white/gold. But after learning it was black/blue I stared at it and the colors seemed to shift in my brain. Now I can't see the white/gold for the life of me.

Same, i simply cannot find any other explanation for it. Apparently a significant amount of the population cannot do white-balancing in their head.
I have to assume they still experience the more physical white-balancing from light depleting the stuff in cone cells though: that thing where if you close one eye for a bit while looking at a sunny landscape, then open it and compare to the other eye by alternating which one you close; you'll see the world tinted.. i think it's blue-green-ish and red-ish depending on the eye. Different tint depending on the eye, at least.