But not everyone agrees on which colors go together and which clash

But perception is for a large part embedded in memory, which differs individually. For me steel foundries smell amazing because I used to play on the beach near a steel foundry, to the point I need to put effort into understanding that it's actually kind of acrid. So am I still "having the same perception" as someone who doesn't have the lived experience?

This can happen at a society-wide level too. Liminal beige and seafoam green were not intended to create a feeling of disquiet, but of calm neutrality. Modern audiences perceive them as disquieting because they have been systematically used in our society to impose a sense of calm on un-calm situations, such as operating rooms or hallways in sketchy buildings.

I honestly don't know how much of the commonality of associations across cultures comes from instinct and how much comes from the fact that all children learn to live on the same planet with the same physical laws. I would bet that for 99.9% of children, their first experience with a strong sulphur smell is going to be from rotten eggs (or similar rotten goods) that others act disgusted by. So the fact that sulphur smells disgusting to the vast majority of adults is not evidence for instinct over memory. The same goes for green plants, red blood, blue skies, etc.

i think the best example of this is music. most people have very good relative pitch perception, but mediocre absolute pitch perception. that shared relative pitch perception though will often be built based on ones own experiences and life. the shared perceptual relativity is both developed socially/culturally and innate based on how we process and abstract information, but that says nothing about our absolute perception of that information.