Mat, thats a very interesting question, which perhaps I would not be able to answer - the best I can do is to recommend to read up on some neurological research publishings, etc.
But I think (without any degree of certainty) that the color is actually matched with correct notes, which is why these people would be the best in terms of perfect pitch. if a note is not played correctly, they will not see the color that they are supposed to see.
Its not always necessarily notes, some people see colors for letters of alphabet, for numbers, etc. Its really quite intersting. I studied things like this a little bit in college. Its a shame I didnt end up doing neurology
P.S. Usually this phenomenon is observed in younger age, but then kind of dissappears little by little, but not in all cases. Sometimes its for life. I think somebody famous (some musician or composer) had that.