Flight instructor
Praise or scold: which one works?
You train twelve pilots. You see a landing that stands out, good or bad, and you say something. Then you watch the pilot's next landing. At the end we count which worked better.
Why scolding looked like it worked
Each landing is skill plus luck. A landing that stands out had more luck than usual, good or bad. Luck does not repeat on order, so the next landing tends to fall back toward the pilot's normal. Great landings get a little worse. Bad ones get a little better. That happens whatever anyone says.
An instructor praises the great landings and scolds the bad ones. So praise is followed by a worse landing, and a telling-off by a better one. The instructor sees punishment work and praise fail, every week, for years. Daniel Kahneman heard exactly this from flight instructors in the Israeli air force. They were sure. The pattern they saw was real. The cause they gave it was not.
The same game in a trading room
Pension plans hire fund managers after a run of great years and fire them after bad ones. Amit Goyal and Sunil Wahal followed 3,400 plan sponsors from 1994 to 2003. The managers they hired had beaten the market before hiring and did not beat it after. Had the plans kept the managers they fired, they would have done no worse than with the new ones.
The great run that gets a manager hired is the stand-out landing. Part of it is luck, and the luck goes home. The committee that fires the loser and hires the winner sees the old manager recover somewhere else, and the new one cool off, and calls it bad luck. It is the most normal thing in the world.
Before you hire on a great run, ask: how much of that run would cool off on its own?
Sources
- Tversky and Kahneman, "Judgment under uncertainty: heuristics and biases", Science 185 (1974), 1124–1131.
- Kahneman, Thinking, Fast and Slow (2011), chapter 17, "Regression to the mean".
- Goyal and Wahal, "The selection and termination of investment management firms by plan sponsors", Journal of Finance 63 (2008), 1805–1847.