I take a cube and a set of identical rulers into a classroom of 5 year olds. I ask each child to measure the cube and write down each of the measurements, in turn.
I then take the rulers into the corridor and sprinkle them with Fairy Dust, in front of all the children.
I then go into the classroom, on the opposite side of the corridor, and ask those 5 year olds to measure the cube again record these measurements.
What has the Fairy Dust done?
-Quite a lot, as it turns out! -Here are the changes in measurement for each ruler, plotted against the initial size measurements:
An apparently highly significant and dramatic effect from the application of Fairy Dust 🙂
OK: So we don’t believe in Fairy Dust and I will tell you that the measurements were generated by Excel (as simply random errors added to the 6 inch cube). So what has happened?
Look at the little inset scatter plot in the top right: Nothing going on, just as suspected. Now look at the larger scatter plot: The most extremely erroneous (and unlikely) large first results (say around 7.5 inches) tend not to be so ‘jammy’ second time around and so tend to be smaller (in fact they tend to be average!). The same thing happens to the most extreme, small, first measurements too.
This is a simple example; one fixed sized cube and you didn’t believe in fairies anyway. But in the real world, outside the office, it is a lot more complicated and quite often WE DO want to believe in stuff and perhaps the changes fit in with our preconceptions.
So where might this occur in practice – where do we have noisy repeated measurements? A possible ‘high risk’ case is soil sampling: you go out one year and take soil samples in your fields you find that you have low and high indices, you apply your fertiliser using the latest tech. You come back a few years later and measure the indices in roughly the same areas again. If you then sort the results by the initial soil Indices, guess what ? -The very high indices have got lower and the lowest indices have got higher.
Your expensive GPS Variable Rate Technology worked (NOT).
And just in case you thought that ‘Statistics’ would automatically sort this out, here are the stats for the effect the fairy dust data:
p=0.0001 ! This analysis, by the way, is generated by ‘PAST’, a very nice standalone stats package, which is free. http://folk.uio.no/ohammer/past/
So: Do not be fooled by plots of change, when the X axis has been sorted.
For a real-world example of the trap have a look at these charts:
If you want to know more; search for “Regression to the mean” rather than ‘reversion to the mean’ as this is the commonest name for the problem.