Then you go and apply S to another population, P. Now things are way off -- the population's clustered around a much lower mean, maybe the distribution looks weird, and so on.
Suppose further that P is a significant part of the total population you're interested in measuring using S. Say, around 12%. And suppose, furthermore, that the mean of P is in the lower third standard deviation of the distribution for R.
Then you have to make a choice between two basic hypotheses: Either individuals in P are significantly lower than individuals in R in terms of whatever it is S is supposed to measure. Or P is a crappy way of measuring this whatever it is -- it's gotten the distribution spectacularly wrong.
How to decide between these two? Well, if you already think that individuals in P are significantly lower than individuals in R in terms of whatever it is S is supposed to measure, then you embrace that hypothesis, and don't really consider whether or not S is working right, because it does indeed appear to be working right.
But then, of course, you can't point to S to argue for the claim that individuals in P are significantly lower than individuals in R in terms of whatever it is S is supposed to measure. That would be blatantly circular.
Back in October, James Watson -- misogynist and Nobel laureate co-discoverer of DNA with Crick and Franklin -- said something rather racist:
[Watson] is "inherently gloomy about the prospect of Africa" because "all our social policies are based on the fact that their intelligence is the same as ours - whereas all the testing says not really", and I know that this "hot potato" is going to be difficult to address.
Pretty indefensible, right? How can anyone think that Africans are less intelligent than `us'? (I'm not even going to get into the assumption that the intersection of `us' and Africans is the empty set.)
Well, thank the internets, someone has risen to Watson's defence.
Watson's claim in his recent interview with Charlotte Hunt-Grubbe that intelligence testing shows lower scores in Africa than Europe is likewise, entirely supported by the scientific literature.
Then there are citations to a bunch of studies that all say about the same thing: on standard IQ tests, the mean throughout Africa is between about 65 and 75. That's apparently right around the current definition of `mental retardation'.
The situation is an instance of the one I described above. Test in Western nations, and you get a mean around 100 (by construction). Test in Africa, and you get a mean in the third standard deviation. Africa's population is about 888 million, which is around 12% of the world's 6 billion. These results are incompatible with a worldwide standard distribution of IQ -- if the standard deviation for the African data sets is the same as the standard deviation for the Western data sets, about twice as many people are in the third standard deviation as should be.
So we have to choose between two hypotheses: either Africans are less intelligent than Westerners (that's highly simplified, but again, bracketing the spectacularly racist assumptions here) or IQ tests are not a good way of measuring intelligence.
There is a long attempt in that post to argue for the negation of the second hypothesis -- that is, that IQ tests are, in fact, a good way of measuring intelligence. The argument comes down to two points: First, IQ test scores correlate with other IQ test scores, and second, IQ test scores correlate with economic success. The first is useless -- the fact that all these tests measure the same thing doesn't imply that they don't all measure the same artificial statistical construction. And the second only implies that IQ tests are a good way of measuring intelligence if economic success is a good way of measuring intelligence. And that's only plausible if you think that rich people are more intelligent than poor people. The possibility that economic status has an effect on IQ scores is never considered.
It all comes down to racism, classism, and the intersection of the two. If you already believed that Africans and poor people are less intelligent than `us', then you won't see any problems here. IQ tests must measure intelligence, because they get the results we expected for intelligence. If you think that the idea that Africans and poor people are less intelligent than `us' is offensive on its face, then you're as liable as before to think that this IQ test business is a load of crap.
Which makes this an excellent example of the way ethico-political values show up as background assumptions when reasoning from evidence to theory. We have the same sets of evidence, but different background assumptions concerning race, class, and intelligence. Based on these assumptions, we reason to very different -- indeed, incompatible -- theories. Without these background assumptions -- if our reasoning process was truly `value-free' -- we would have no way of reaching any conclusions beyond statistical correlations between different metrics.
However, where I am willing to embrace the role my values play in my reasoning, the defender of Watson, it seems, does not.