Choice & Inference

Jonah Lehrer published in the New Yorker an intriguing article about the “decline effect” which is the tendency of many exciting empirical results in science to fade over time.  After receiving many letters and responses he recently published some afterthoughts in the blog of the New Yorker. Perhaps one of the most interesting responses is the one offered here by Andrew Gelman a professor of statistics at Columbia. I think that I have seen various instances of the decline effect in behavioral decision theory.  But these cases  might be just examples of “selective reporting” or instances of the phenomenon that Richard Feynman describes in his Commencement Address at Caltech in 1974:

Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It’s a little bit off, because he had the incorrect value for the viscosity of air. It’s interesting to look at the history of measurements of the charge of the electron, after Millikan. If you plot them as a function of time, you find that one is a little bigger than Millikan’s, and the next one’s a little bit bigger than that, and the next one’s a little bit bigger than that, until finally they settle down to a number which is higher.

Why didn’t they discover that the new number was higher right away? It’s a thing that scientists are ashamed of—this history—because it’s apparent that people did things like this: When they got a number that was too high above Millikan’s, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number closer to Millikan’s value they didn’t look so hard. And so they eliminated the numbers that were too far off, and did other things like that.

But if Gelman is right there might be cases of the decline effect that could be more interesting.  Any thoughts?