I can see one way we might be able to test this.
This idea, of course, rests on you first having accepted that what we subjectively experience is a product of the brain itself (you might get away with just saying that our perceptions of/from the senses, at least, are a product of our brain) and how the various cells of the brain communicate, what patterns they use, and the timing of their signalling. Using that starting point, you can see that what you "see" is highly dependent upon the structure of your brain, and the unique way that it has developed over your life due to experience and your own genetics.
Your mind has to take the various signals it gets from your retina, and weave them into a consistent and perceptible whole for you to subjectively experience. There is no such thing as red, yellow, or blue, at least, not inherently within the universe itself. All colours, indeed, the entire electromagnetic's range, are just different wavelengths of light/photons. To the universe itself, nothing has "colour", just different photons' wavelengths.
Edit (meant to include this):
Our retinas take information about the wavelengths of light, transmit it through the optic nerve to the brain, the brain interprets this information, and using a group or group of groups of cells somewhere in the visual processing centres of the brain, ties our unique and personalized sense of colour into our general subjective consciousness.
So what we perceive as colour is really just heavily processed and edited information from our retinas which had coded the wavelength of the light. This information then enters into the parts of the brain that handle subjective awareness of sight. Sight already ties in information from separate groups of neurons in different parts of the brain, such as parts processing shape, size, depth, etc. It also ties in colour.
Since this is how colour is produced, and each brain is different in neuron location, amount, pattern, etc., then I find it likely that we do see different colours. Each brain being slightly different, how the visual information gets tied into subjective consciousness would not be exactly the same. There may be similarities, and undoubtedly some near matches, but probably everyone would be seeing different colours subjectively, while still calling them the same.
In partial support for this, it is definitely already true that men and women do see colours differently, with the average woman being able to see far more wavelengths (read colours) than the average man. So their brain has a more "advanced" visual processing system in regard to colour, which, as far as I know, comes from partial activity of a degraded fourth cone. Cones are cells that perceive colour in our eyes, as opposed to rods which can only provide brightness, not colour. On average, their eyes see far more colours than average, and there are an unknown number of females, likely millions, if not more, who can see even more colours due to highly functioning fourth cone cells; cells that are always inactive in males.
Interestingly, the same dichotomy appears in monkeys. Old World Monkeys have colour vision, using the same cones we do. In fact, all living Old World primates see colour with this system (though all have different brains, and likely see the different "colours" from the same wavelengths), and very few New World monkeys do. This is because most mammals only have two cones, not three like primates. All other land vertebrates have it, such as birds and reptiles. Why not mammals? Because we lost our fourth cell during the reign of the dinosaurs, when the mammals that led to us all were nocturnal in order to survive. Primates managed to re-evolve it, but only after the split between New and Old World Monkeys (really cool story how they got there, too, it involves a huge raft of debris bringing ancient monkeys across the smaller Atlantic to S. America).
Luckily for a select group of monkeys in S. America, they, too managed to re-evolve the fourth cone. What's interesting is that only their females carry this trait, never the males. This is because the gene for this is carried on the X chromosome, and you have to have two of them to get the trait of being able to see an estimated 100 million colours subjectively, compared to 1 million for the males.
There's no need to see the same "colours" subjectively in different brains just a need to associate the same groups of wavelengths with the appropriate response or linguistic concept.
Here's a neat article about females seeing these wavelenths/colours:
The Humans With Super Human Vision | DiscoverMagazine.com
Excerpt:
Each of the three standard color-detecting cones in the retina -- blue, green and red -- can pick up about 100 different gradations of color, Dr. Neitz estimated. But the brain can combine those variations exponentially, he said, so that the average person can distinguish about 1 million different hues.
A true tetrachromat has another type of cone in between the red and green -- somewhere in the orange range -- and its 100 shades theoretically would allow her to see 100 million different colors.
That may be why Mrs. Hogan can look out the windows of her Mount Washington home and tell the relative depths and silting of the three rivers at the Point by discerning the subtle differences in their shades.
"I have a very hard time even giving names to colors because I see so many other colors inside them," she said.
Dr. Neitz, who conducts his research with his wife Maureen, said only women have the potential for super color vision.
That's because the genes for the pigments in green and red cones lie on the X chromosome, and only women have two X chromosomes, creating the opportunity for one type of red cone to be activated on one X chromosome and the other type of red cone on the other one. In a few cases, women may have two distinct green cones on either X chromosome.
Read more:
Some women may see 100 million colors, thanks to their genes - Pittsburgh Post-Gazette
He suggests that there may be 99 million women with this trait. They clearly see different colours than males or other females do.
Add this with things like synesthesia and other products of differing systems of visual processing cells and patterns, and I think you've got a strong hypothesis for subjective visual perception of qualitatively different "colours" from the same wavelengths being likely for most, if not all people.
How much it differs, and when, are the questions, I'd think. It probably wouldn't be by much, in most cases, but there would be at least some differences. Still, this is just my hypothesis. I know I'm probably wrong at least somewhere.
