You can literally sample the rgb values

It doesn't matter. This phenomenon can be explained by something called color constancy.

I remember some versions of this image where I could literally switch between perceptions at will, when I imagined different surrounding light temperatures/environments.

It's a subjective perception.