# Diagnosis of Forecasting Errors

There are infinitely many ways to examine forecast errors to learn more about them and perhaps correct future forecasts for their deficiencies. In this section we first present a diagrammatic scheme that can be used to learn a great deal about the pattern of forecast errors, and then we present some numerical techniques for computing diagnostics.

Graphical Analysis One of the simplest, and yet most revealing, techniques for examining the pattern in forecast errors is the Prediction Realization Diagram (PRD) proposed by Thiel [11, 12], This diagram is simply the plot of the predicted change in earnings against the realized change. The predicted change is plotted along the line that lies at a 45-degree angle to the horizontal axis, and actual change is plotted along a line that lies at a minus 45-degree angle to the horizontal axis. This is shown in Figure 25.1.

If we plot in this space the forecasted change in earnings versus the realized change, we can learn quite a lot about the type of forecast errors being made. Notice that if a point lies on the horizontal straight line, it indicates that the forecast change was exactly equal to the actual change. To the extent that points lie above the horizontal straight line, it indicates that estimates were too high. To the extent that points lie below the horizontal line, it indicates that estimates were too low. Now let us take a closer look at what each section of the graph represents. A point lying in section I of the PRD indicates that the forecaster successfully predicted that earnings would increase, but that the size of the increase was overestimated. A point lying in section II indicates that the analyst successfully predicted a

'When earnings are zero, the percentage change in earnings will be infinite. When earnings are negative, the meaning of percentage changes in earnings is ambiguous.

decrease in earnings but that the size of the decrease was underestimated (earnings were overestimated). If a point lies in section V, it indicates that the analyst predicted the wrong direction for the change in earnings. That is, the analyst predicted they would increase when they, in fact, decreased. Sections HI, IV, and VI are analogous to sections I, H, and V. Section HI represents a successful prediction of an increase in earnings but an underestimate of the size of the increase. Section IV represents a successful prediction of a decrease but an overestimate of the size of the decrease. Finally, a point in section VI represents a forecast of a decrease in earnings when they actually increased. Sections V and VI indicate that the analyst misestimated the direction of the change in earnings movements whereas the other sections indicate the analyst got the direction right but the size wrong.

Examination of a group of forecasts on the PRD can reveal quite a lot of information about the source of error in analysts' forecasts. In Figures 25.2 and 25.3 we have constructed two hypothetical patterns that might be observed. Figure 25.2 presents the case where a forecaster is consistently optimistic. The forecaster consistently overestimates earnings changes when they are positive (section I) and consistently underestimates the size of a negative change (section II) or actually predicts a positive change when changes are negative (section V).

Figure 25.2 Prediction Realization Diagram: optimistic forecaster.

P;-R;

Figure 25.3 Prediction Realization Diagram: the overreactor.

A more interesting pattern is revealed in Figure 25.3. In this diagram we find the profile of an analyst who is excellent at predicting the direction of change. However, the analyst overreacts to change by becoming overoptimistic as large positive changes are expected and overpessimistic as large negative changes are expected. This can be seen by the fact that estimates lie further and further from the horizontal axis as the actual level of earnings change increases.

These are only two of the many potential patterns of forecaster behavior that can be seen through the PRD. The reader is encouraged to construct other series of points on this diagram and to interpret their meaning.

Numerical Analysis While the graphical analysis of forecast errors is extremely useful, there are also several analytical decompositions of the mean squared forecast error that can provide useful insight into the sources of forecasting error. Let us take, as an example, a firm that has a collection of analysts making forecasts of a broad group of stocks in the economy. We discuss two meaningful decompositions of these errors. The first is based on the level of aggregation at which errors occur, while the second looks at the forecast error in terms of the characteristics of the forecasters.

Error Decomposed by Level of Aggregation It would be extremely useful to determine at what level of aggregation errors in the forecasts occurred. One scheme for analyzing the level separates earnings errors into three components. This scheme is shown below.

where

P = mean value of P; across all stocks followed by all analysts R = mean value of Rj across all stocks followed by all analysts

Pa = mean value of P, for industry a to which i belongs. Each industry will have a different value of Pa

Ra = mean value of for each industry in turn

The first term measures how much of the forecast error is due to inability of the analysts, in total, to predict what average earning will be for the economy. This term is simply the squared difference between the average predicted change in earnings and the average realized change in earnings. The second term is a measure of how much of the total error is due to the individual analysts misestimating the differential performance of particular industries from the average for the economy.

Let us examine this term in more detail. For each firm (i) the difference between the mean predicted change in earnings for the industry to which it belongs (PJ and the mean predicted change in earnings for all firms (P) is calculated. The same term is calculated for actual change in earnings (Ra - R). The difference between the two is squared and summed for all firms. Then the average value of this term is taken. The third term measures how much of the error is due to analysts not being able to predict the difference in performance of the individual stocks they follow from the appropriate industry average. The first part of this term is the difference between the predicted change in earnings for an individual stock Pt and the average predicted change for a stock in the industry to which i belongs. The second part has the same meaning, but deals with realizations. These differences are squared, summed, and then averaged.

By dividing through both sides of the equation by the MSFE, we express each source of error as a fraction of the total mean squared forecast error. Diagnosing the source of the error can be of great significance to the firm. For example, if the major source of error arises from misestimated aggregate earnings, then the company should concentrate more effort on preparing its forecasts of the general economy. If the analysts are provided with better information about the aggregate level of earnings and are explicitly encouraged to use this information, then improvement should occur in the firm's forecasting effort. Assuming that each analyst follows one industry or a group of closely related industries (economic sector), then a large value for the second term points to an error in understanding the economics of alternative industries. Large values of the third term indicate that errors are associated with being unable to differentiate between the performance of individual companies even when mistakes in forecasts of the level of the economy and industries are removed.

Obviously, this same type of decomposition can be repeated for all individual analysts with their error decomposed into their misestimate of how the stocks they follow will do, on average, and their inability to differentiate the performance of the companies they follow.

In Table 25.2 we present the decomposition of the mean square error by level of aggregation for the set of consensus forecasts discussed earlier. Perhaps the most striking aspect of this table is the small percentage of error that is due to misestimating the performance of the economy (the average company). This source of error never exceeds 3% of the total mean squared error. The percentage of error due to industry misestimates starts at 36.5% in January and declines continuously to 17.6% by the end of the fiscal year. Consequently, the error due to misestimating individual companies grows from 61.8% to 80.5% over the year. We have already seen that the size of analyst's errors shrinks over the year. Now we see that while analysts become more accurate in forecasting both industry and company errors, their ability to forecast industry influences grows relative to their ability to forecast company performance over the year.