Good decisions are expensive

Posted on April 12, 2018

I think a lot of discourse tends to be off the mark because the immense difficulty of making good, high-quality decisions isn’t appreciated.

My favorite example for this is borrowed from Kahneman’s book Thinking Fast and Slow. He introduced it to make a different point, but what I got out of it is that even in some of the simplest circumstances, being correct is so difficult and tricky that it’s unreasonable to expect it from anyone.

This scenario is this. There’s an military instructor, teaching students to fly airplanes. As part of the training, he goes up with the students on test flights. Sometimes they do well, sometimes they do poorly, and he gives them praise or criticism appropriately.

So far, so good. Now, after a while, being very careful and observant, he starts to notice a pattern.

Whenever he gives praise, the students do worse afterwards. And whenever he criticizes them, they improve. And this pattern just won’t go away, even after he starts writing it all down at tracking it.

So, in the end, after a lot of thought, he reaches the only reasonable conclusion: praise hurts the students performance, and criticism helps it. So the correct thing to do is to stop praising students, and ramp up the criticism.

The only problem is that this is 100% completely dead wrong. If you haven’t heard this one before, I encourage you to try to do better and figure out why.

Before we get into the answer, let’s unpack the situation.

  • this instructor is a well-qualified expert in his field
  • this is really important to him
  • it’s actually a very simple scenario
  • he’s super careful
  • he’s thinking as hard as he can
  • his observations are correct. students really do get worse after praise, and better after criticism. He’s just wrong about why.

On the whole, I don’t fault this instructor in the slightest for getting this one wrong. He’s not dumb or lazy or uninterested or biased or trying something too ambitious. Making correct decisions is just Really Hard.

So, here’s what’s really going on - regression to the mean. Basically, how well a student flies on any particular day is a combination of skill and luck (maybe 30/70, 50/50, 95/5 - doesn’t matter for our purposes).

When the instructor gives out praise, it’s after really good performances. The best performances will be, for the most part, lucky. And likewise criticism will be given after performances that are more unlucky than usual.

The key characteristic of luck though, is that it doesn’t stick around. The next day, when the student goes up in the air again, they’ll have the same skill as before, but on average they’ll have average luck. If they were lucky the day before (and got praised for it), they’ll do worse, because average luck is worse than good luck. If they were unlucky the day before (and got criticized for it), they’ll do better, because average luck is better than bad luck.

There’s more to say on it, but that’s the overview and there’s no point in me parroting the whole wikipedia article. (Interesting tidbit though - this is also how placebos “work” in medicine).

For our purposes, the important thing here is that it’s not the instructors fault that he doesn’t know about this obscure statistical trap. So unless and until we live in a world where people consult their local statisticians before daring to say “good job out there today”, we have to recognize that decisions are difficult.