Even if it did spit out the average value, it would be the average of the training data. I don't think people/opinions are evenly distributed in LLM training data.

Just look into how racist computer vision models can be.