clipped from www.newyorker.comThe Truth Wears Off
Is there something wrong with the scientific method?by Jonah Lehrer December 13, 2010
Jonah Lehrer, Annals of Science, "The Truth Wears Off," The New Yorker, December 13, 2010, p. 52
ABSTRACT: ANNALS OF SCIENCE about the decline effect.
On September 18, 2007, a few dozen neuroscientists, psychiatrists, and drug-company executives gathered in a hotel conference room in Brussels to hear some startling news. It had to do with a class of drugs known as atypical or second-generation antipsychotics, which came on the market in the early nineties. The therapeutic power of the drugs appeared to be steadily falling. A recent study showed an effect that was less than half of that documented in the first trials, in the early nineties.
Before the effectiveness of a drug can be confirmed, it must be tested again and again. The test of replicability, as it's known, is the foundation of modern research. It's a safeguard for the creep of subjectivity. But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It's as if our facts are losing their truth.
This phenomenon doesn't yet have an official name, but it's occurring across a wide range of fields, from psychology to ecology.
When Jonathan Schooler was a graduate student at the University of Washington, he discovered a surprising phenomenon having to do with language and memory that he called verbal overshadowing. While Schooler was publishing his results in journals, he noticed that it was proving difficult to replicate his earlier findings. Mentions psychologist Joseph Banks Rhine, who conducted several experiments dealing with E.S.P. In 2004, Schooler embarked on an imitation of Rhine's research in an attempt to test the decline effect.
The most likely explanation for the decline is an obvious one: regression to the mean. Yet the effect's ubiquity seems to violate the laws of statistics. Describes Anders Møller's discovery of the theory of fluctuating asymmetry in sexual selection. Mentions Leigh Simmons and Theodore Sterling.
Biologist Michael Jennions argues that the decline effect is largely a product of publication bias.
Biologist Richard Palmer suspects that an equally significant issue is the selective reporting of results—that is, the subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results. Mentions John Ioannidis.
In the late nineteen-nineties, neuroscientist John Crabbe investigated the impact of unknown chance events on the test of replicability. The disturbing implication of his study is that a lot of extraordinary scientific data is nothing but noise. This suggests that the decline effect is actually a decline of illusion.
Many scientific theories continue to be considered true even after failing numerous experimental tests. The decline effect is troubling because it reminds us how difficult it is to prove anything.
On September 18, 2007, a few dozen neuroscientists, psychiatrists, and drug-company executives gathered in a hotel conference room in Brussels to hear some startling news. It had to do with a class of drugs known as atypical or second-generation antipsychotics, which came on the market in the early nineties. The therapeutic power of the drugs appeared to be steadily falling. A recent study showed an effect that was less than half of that documented in the first trials, in the early nineties.
Before the effectiveness of a drug can be confirmed, it must be tested again and again. The test of replicability, as it's known, is the foundation of modern research. It's a safeguard for the creep of subjectivity. But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It's as if our facts are losing their truth.
This phenomenon doesn't yet have an official name, but it's occurring across a wide range of fields, from psychology to ecology.
When Jonathan Schooler was a graduate student at the University of Washington, he discovered a surprising phenomenon having to do with language and memory that he called verbal overshadowing. While Schooler was publishing his results in journals, he noticed that it was proving difficult to replicate his earlier findings. Mentions psychologist Joseph Banks Rhine, who conducted several experiments dealing with E.S.P. In 2004, Schooler embarked on an imitation of Rhine's research in an attempt to test the decline effect.
The most likely explanation for the decline is an obvious one: regression to the mean. Yet the effect's ubiquity seems to violate the laws of statistics. Describes Anders Møller's discovery of the theory of fluctuating asymmetry in sexual selection. Mentions Leigh Simmons and Theodore Sterling.
Biologist Michael Jennions argues that the decline effect is largely a product of publication bias.
Biologist Richard Palmer suspects that an equally significant issue is the selective reporting of results—that is, the subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results. Mentions John Ioannidis.
In the late nineteen-nineties, neuroscientist John Crabbe investigated the impact of unknown chance events on the test of replicability. The disturbing implication of his study is that a lot of extraordinary scientific data is nothing but noise. This suggests that the decline effect is actually a decline of illusion.
Many scientific theories continue to be considered true even after failing numerous experimental tests. The decline effect is troubling because it reminds us how difficult it is to prove anything.
No comments:
Post a Comment