Imperfect prediction
So far we have discussed two cases of prediction:
(a) a dichotomous predictor (e.g., symptom) with a dichotomous outcome (e.g.,
illness), for which we used 2x2 tables, conditional probabilities, and Bayes
theorem;
(b) a continuous predictor (e.g., perception) with a dichotomous outcome (e.g.,
person likes me/doesn't like me), for which we used signal detection
theory.
Now we add a third case, in which both predictor and outcome are continuous. For example, you know a person's position in the first heat of a ski race, and you want to predict that person's position after the second heat. Or you know a patient's response to therapy after 4 weeks and want to know his or her response after the full 8 weeks.
The most elegant formal model of this third case of prediction is regression analysis, and it shows very clearly how prediction virtually always implies making mistakes. In the discussion of Bayes' theorem and SDT we saw that dichotomous prediction implies being wrong a certain proportion of the cases (e.g., your false alarm rate). In regression analysis, mistakes are typically called "error of prediction"--the distribution of true values around your predicted value. In the figure below (from your flower naming experiment) you see that even at a correlation of r = 0.85 these errors of prediction (the little distributions around the regression line) are substantial. That is, if we predict a particular performance (say, from an estimate of 10), we need to consider that the true value will lie within a range called our "confidence interval," which is built around the predicted value (here, 9)
A central aspect of imperfect prediction is "regression to the mean." For
example, if you try to predict one class grade from another class grade, you
typically find correlations in the .3 to .6 range (see figure below). This
implies that people who did well in the first class (e.g., 25) will tend to, on
average, still do okay in the second class, but many of them will do worse
(10). On the other hand, people who did badly in the first class (e.g., 5)
will do better in the second class (10). Of the people in the middle some will
do better, some will do worse. You can still make predictions: For each grade
of the first class you can make a best guess as to the grade in the second
class. But these best guesses lie on a regression line that is relatively
flat, so you cannot predict extreme scores on the outcome (even if the
predictor had an extreme score).
Typical examples of regression to the mean include: predicting sport results
(e.g., final results from the first heat; play-off performance from regular
season), business outcomes (e.g., stock or sales trends from one year to the
next), social behavior (from one situation to the next), or therapy (e.g.,
comparing the first half of a treatment to the second half). In all these
cases, many causal links might exist between one measurement and the next.
Whether or not there are causal links, however, the best people or cases on the
first measurement will be worse on the second one, and the worst on the first
will be better on the second. So if your therapy doesn't go so well in the
first half, chances are it will get better in the second half; if a team was
particularly good in the regular season, chances are it will not be quite as
dominant in the play-offs, etc. All you need to know for these insights is
that there is an imperfect correlation between the two time points--if that
holds, regression to the mean will occur.
Overconfidence and accuracy
Overconfidence is an unjustified conviction that your prediction is correct. How do we know whether the strength of a conviction is unjustified? Simply being wrong after having made a prediction with 80% confidence does not give us the necessary data (because it's okay to be wrong sometimes when you are 80% confident). Somebody is overconfident if the person's 80%-confident predictions are correct in fewer than 80% of all cases. That is, if I make 10 predictions with 80% confidence, 8 out of those 10 must be correct. (Tests like these are known as "calibration".)Overconfidence is "undesirable" because it creates false security (or false fear). Imagine how quickly you would lose trust in a doctor's or financial advisor's opinion if she or he were overconfident.
Research has identified three main factors that increase overconfidence:
- Unfamiliar domains. The less we know about a domain, the less accurate we will be. Unfortunately, our confidence does not decrease proportionally. As a result, we will be particularly overconfident in domains where we don't know much (cf. the class exercise)
- Greater confidence. The greater people's confidence, the more overconfident they are. So beware of extremely confident predictions--they are likely to be overconfident.
- Going against base rates. If an exemplar (e.g., a person) looks particularly representative of a category, and if that category is rare (low base rates), then people are likely to overestimate the likelihood that the person belongs to the category and they are likely to be overconfident in their prediction (because it seems so right...). A prediction that goes against base rates (i.e., predicting that an exemplar belongs to a rare category) is therefore doubly risky: you are likely to be wrong, and you are likely to be overconfident.
Experts are at least as overconfident as non-experts; sometimes even more (because they think that they, as experts, are more likely to be correct). The only experts who fare well on calibrations are weather forecasters. Why? because they receive continuous feedback on the justification of their confidence levels.
How can you avoid overconfidence? By avoiding predictions in unfamiliar domains; by adjusting your own confidence downwards, especially if it gets high; and by actively challenging your own beliefs (thinking of reasons why you might be wrong, trying to falsify your assumptions).
Prediction from multiple data
Whenever you make predictions from several pieces of information, you face a "combination problem": How should you combine the pieces in order to maximize your predictive accuracy?Two methods exist:
- In a global judgment you look at all the information simultaneously and "simply make" a prediction. The problem with this method is that it is a "one-item test," hence very unreliable. Today you might have one global impression, tomorrow you might have a different one. If you are only a little bit biased, your prediction will be off--because of vivid pieces of information, prejudices, or lapses of attention that made you overlook crucial information.
- In a component analysis you look at each piece of information separately and decide whether it is positively (a score of +1) or negatively (a score of -1) associated with the outcome you want to predict. This method therefore forces you to pay detailed attention to each piece. Adding up all the +1 and -1 values, you then get an aggregate score that has all the advantages of a multi-item test--e.g., it is more reliable; it is less influenced by biased perception of individual (e.g., vivid) pieces.
In hundreds of studies across numerous domains (e.g., job hiring, school admissions, clinical diagnosis), component analysis has consistently fared much better than global judgment. Nevertheless, and unfortunately, many people still seem to prefer global judgments--perhaps because they are overconfident in their judgment capabilities, perhaps because it is "easier" (i.e., less work) to simply eyeball a large amount of information than to pay attention to each piece individually.