Verified by 
Note: The ideas for this essay emerged from the Scientific Self-Correction working group of the Society for the Improvement of Psychological Science. A followup blog will introduce this new organization working to, shockingly, improve psychological science in more detail.
Contributors to the ideas presented here include Lisa Auster-Gussman, Andy Vonasch, Marcel van Assen, Richie Lenne, Joe Hilgard, Victor Keller, Traci Mann, Alex McDiarmid, Alexa Tullett, and Simon Columbus. Nonetheless, all errors, outrages, misstatements, and other problems with this essay are entirely mine.
Why You Should Care About Whether Psychology is a Self-Correcting Science
First, for you, my non-expert audience. You know all those great advice columns, with titles like (I have made these up, but it is easy to find things like these): Six ways to tell if your partner is a narcissist; Four ways to have better sex; What really counts for teens?
The information in such essays are usually either true or untrue; sometimes they can be oversimplifications that are misleading. Scientific research, if it exists on the topic, is almost always the best way to determine whether something is true, not true or kinda true sometimes. But there is a problem. Scientific “facts,” are not quite the permanent, unchangeable things they seem to be. In 1900, scientists had concluded that the Sun was made of the same stuff as the Earth. By 1950, it was well-understood that the Sun was mostly hydrogen and helium. In 1980, the best minds in medical science had concluded that stress causes ulcers; in 2005, Barry Marshall received the Nobel Prize in medicine for showing that bacteria causes ulcers (although stress can make tissues more vulnerable to bacterial infections).
Why is this important? One of science’s great claims to special status and credibility is that it is “self-correcting” – i.e., that when new data comes in showing something that conflicts with scientific conclusions, scientists supposedly change their conclusions.
Except that, too often, they do not, at least not very easily. The accepted scientific wisdom does change, but usually because new scholars take up the correction, not because the old guard “self-corrects.” Think I am either joking? Then consider what the great Nobel laureate and nuclear physicist, Max Planck, once said: “A scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it.”
This means that you, consumers of psychological science, may be routinely “sold” claims that are dubious at best, and completely false at worst. Think I am kidding? You may have run across some of these claims:
1. Power posing can change your life
2. Stereotype threat explains racial differences in academic achievement
3. Implicit biases of which we are not aware are a powerful source of discrimination and real world inequalities
4. Smiling improves your mood
5. Feelings of disgust influence moral judgments.
6. Conservatives deny science a lot more than do liberals
7. Stereotypes are inaccurate
8. People explain their own behavior as caused by their situations; they explain others’ behavior as caused by their personal characteristics.
9. Diversity programs are highly effective at reducing discrimination
10. Microaggressions are a well-established and pervasive form of discrimination
11. Type A behavior is bad for your health
All of these claims have either been subject to failed replications or have proven themselves dubious, characterized by inconclusive evidence, or controversial among scientists.
And this does not even include some large proportion of the self-help advice promoted in books and blogs.
Why and How are Dubious Claims Promoted?
1. Honest errors. Sometimes, evidence seems to support a claim, and subsequent research identifies problems with the original studies or demonstrates that the finding cannot be replicated. Science, even at its best, can resemble semi-systematic trial and error. And that means that sometimes, findings produced honestly and even with the best methods may sometimes turn out to be incorrect.
2. Premature promotion of dramatic findings. The problem of honest errors would be less serious if scientists did not rush out to promote them prior to extensive skeptical vetting by the scientific community. Such vetting can often take a decade or more. This may sound like a long time, but consider this timetable:
Year 0. Some study gets published. It takes a few years for people to know about it and digest its importance.
Years 2 and 3. Others begin to design replications and skeptical examinations of the claims.
Years 4 and 5. Those first follow-up studies begin to get published.
Years 6 and 7. The follow-ups begin to disseminate through the field.
Years 8 and beyond. A fuller understanding of the original phenomenon and its limitations begin to emerge.
Why, then, do so many psychologists rush to promote their findings in blogs, press releases, major newspapers, and TED talks before they have been fully vetted? Do you really have to ask? First, there are all the usual self-serving reasons. If Dr. X can make a name for him or herself with some dramatic finding, get grants and tenure, receive large consulting fees and write a bestselling book by the time the rest of the world discovers that X’s original finding ain’t what it’s cracked up to be, X is a named chair at a major university. If anyone complains about this, X can then fall back on, “Well, that’s how science works. We do not always get it right the first time. One of the great strengths of psychological science is that it self-corrects.”
However, more often, I suspect that psychologists do this in all earnestness. They mean well, they are excited by their findings, they believe they have found something important, and they want to tell the world. But, ultimately, with respect to self-correction, it does not matter much whether scientists leap to conclusions and rush to tell the world for good or bad reasons. Either way, a healthy dose of self-correction will be necessary.
3. The compelling narrative. Scientists strive to write well and some succeed brilliantly. Writing well is a crucial skill for communicating to both other scientists and the lay public. Nonetheless, most of us were taught to write compelling narratives, rather than acknowledge the messy or inconclusive truth of most data. In fact, this probably created a disconnect between the strength of the narratives and the strength of the evidence, at least in my home field of social psychology. Many studies used such small samples that they all should have been viewed as tentative pending replication.
4. Refusal to accept disconfirmation. The number of times that critics find errors in papers, and the original authors respond with some variation of, “Thanks for the correction; of course, our main findings still stand” is legion. Another very often heard reply is “[Thanks but] the replication failure merely points out that our finding, which still stands, does not generalize to other contexts”. Or the suggestion that the scientists conducting the replication fail to carry out the research well.
Just recently, this exchange occurred on the NPR program, Radiolab, which did a segment on stereotype threat, a controversial area of research, in which its advocates claim that fear of confirming negative stereotypes about race or sex undermines the academic achievement of African Americans and women. About 32 minutes in, you will hear this:
Radiolab: "Given that replication has been a conversation that you have to unfortunately (sic) contend with, [has] it changed your opinion of the work?"
Claude Steele (the originator of the concept of stereotype threat): "No. I don't think there is anything that could make me think this whole thing is not true."
Well then. If evidence cannot change a scientific claim, what can?
Epilogue on Scientific Self-Correction and Stereotype Threat
Shortly after posting this, I discovered that Claude Steele posted an essay that, depending on one's interpretation, can be viewed as either distancing himself from that remark or explaining why, at the end of the day, he was justified. You can find the whole essay here.
Because the purpose of this post was on the general issues of scientific self-correction, I will leave this issue here for now.
----
I have a follow up essay on efforts to improve psychological science here.
If you find this essay interesting, you can follow me on Twitter, where I focus on issues of stereotype, prejudice, and discrimination, science reform, and speech and academic freedom.
