Confidence and illusion in education

An excerpt from Daniel Kahneman’s forthcoming book Thinking, Fast and Slow was published today in the New York Times Magazine under the title “Don’t Blink! The Hazards of Confidence.” The article has applications to the current discussion about education.

Daniel Kahneman, emeritus professor of psychology and of public affairs at Princeton University and a winner of the 2002 Noble Prize in Economics, tells about his personal experience evaluating the leadership potential of candidates for army officer training.

The evaluators’ rigorous methods consistently failed to select candidates that the commanders at the training school viewed as officer material. Despite that regular negative feedback, Kahneman and his colleagues continued to hold confidently to a belief in the validity of their predictions.

He says their error in attempting to predict behavior from a short artificial situation is a common fallacy into which people slide when faced with a difficult situation. “We are prone to think that the world is more regular and predictable than it really is,” Kahneman says.

That’s why people who are gung-ho about using tests to predict students’ future behavior in totally different real life situations are willing to believe in the validity of those tests even in the face of evidence to the contrary.

Before the anti-test folks start to crow, they might want to read the final paragraph of the piece. In it Kahneman talks about factors that lead to development of what we might call “gut-feeling expertise”: the ability to accurately intuit a judgment. That ability, Kahneman says, is developed from “prolonged experience with good feedback on mistakes.”

Two factors figure into such experience, he says.  First the environment needs to be regular so the observations are not merely anecdotal. Second is “the professionals’ experience and on the quality and speed with which they discover their mistakes.”

Those two factors suggest reasons the confidently expressed observations of the educator can be as flawed as the scores of the standardized test. Classrooms are not noted for their regularity, and mistakes made by teachers may not show up for years, perhaps decades.

In general, Kahneman says:

you should not take assertive and confident people at their own evaluation unless you have independent reason to believe that they know what they are talking about. Unfortunately, this advice is difficult to follow: overconfident professionals sincerely believe they have expertise, act as experts and look like experts. You will have to struggle to remind yourself that they may be in the grip of an illusion.