Introduction of "Prediction Error" as a Measure of the Accuracy of EPDs

David Notter

Department of Animal and Poultry Sciences

Virginia Tech

The 2001 NSIP genetic evaluations introduce a new measure of the accuracy of EPDs. The measure is known as "prediction error" and is designed to avoid some of the confusion and misunderstanding that has been associated with the accuracy measures used in the past. The prediction error directly reflects the amount of future change in EPD that can be anticipated as more data accumulates on an animal, its relatives, and, most importantly, its progeny. Prediction error is expressed in the same units as the trait being evaluated (pounds for weight traits, microns for fleece grade, etc.). Thus each animal has a prediction error associated with the EPD for each trait.

Recall that the EPD is an estimate of its genetic merit based on accumulated performance records. As more information accumulates, the EPD can change. When little information is initially available, future changes can be relatively large. But once a substantial amount of performance data accumulates, the EPD becomes increasingly stable. The prediction error is a measure of the anticipated stability of an EPD. It differs from the accuracy values used before in that it directly addresses the magnitude of possible future change in the EPD whereas the accuracy gave only a relative measure of stability of the EPD. The prediction error was used for a time in the beef industry, where it was known as the "possible change", but was eventually discarded in favor of accuracy. In NSIP, where accuracy values are lower than in the beef industry, we believe that prediction error is a more useful measure of the stability of the EPD.

An EPD for an animal can be though of as an estimate surrounded by error. The prediction error quantifies the magnitude of that error. The properties of prediction error can be summarized relatively easily:

Figure 1 shows the probability that the true EPD will lie within the interval defined by the reported EPD plus or minus some multiple of the prediction error. For example, the true EPD is expected to be within the interval defined by the reported EPD, plus or minus .5 times the prediction error, about 40% of the time. As discussed above, the true EPD is expected to be within the interval defined by the reported EPD, plus or minus 1.0 times the prediction error about 67% of the time. The true EPD is expected to be within the interval defined by the reported EPD, plus or minus 1.5 times the prediction error over 80% of the time.

It is important to realize that the reported EPD is the best estimate we have of the true EPD, and the most likely value of the true EPD. When possible change is large, future changes in EPD may be relatively large. However, it is also important to recognize that the direction of these changes is not predictable. An animal with a large positive EPD and high possible change value could show a significant future drop in its EPD. Or its EPD could go up substantially. Either result is equally likely, which is why breeders should focus on the reported EPDs, and largely ignore accuracy and prediction error, when attempting to determine relative genetic merit. Prediction error should be used only as a risk management tool. If two rams being considered for use have similar EPDs but differ substantially in prediction error, the ram with the smaller prediction error is the less risky choice, since his EPDs are more stable. Prediction error alone is of no value; a ram with mediocre EPDs and low prediction errors is simply an animal that you can be confident will be mediocre.

In order to assist with comparisons between the accuracy and the prediction error, Figure 2 shows the relationship between the two measures of reliability of the EPD for each trait. The maximum value for prediction error should not exceed the value shown for an accuracy value of zero. The only exception to this rule if for inbred animals. If an animal is inbred, its prediction errors will be somewhat larger than those for noninbred animals.

Table 1 shows examples of EPDs, prediction errors, and accuracy values for several types of animal drawn from this year’s genetic evaluation. The animals chosen include:

A) A trait-leader for 120-day weight

B) The ram with the highest accuracy (lowest possible change) for weaning weight

C) A ram lamb in the top 5% for weaning weight EPD

D) A 5-year-old ewe in the top 5% for percent lamb crop EPD

E) A yearling ewe in the top 5% for maternal milk

F) A ewe lamb in the top 5% for percent lamb crop EPD

Note that there is a direct correspondence between the values for accuracy and prediction error: within a trait, high values for accuracy are associated with smaller prediction errors.

Animal A is a high-growth sire with substantial numbers of progeny. For this sire, accuracies are relatively high for growth traits and prediction errors are substantially below EPDs for these traits, indicating that the genetic superiority suggested by the EPDs is real and unlikely to be lost as more information accumulates. For animal A, the weaning weight and 120-day weight EPDs are 2.2 and 2.0 times their prediction error. These values suggest that the probability that this animal's true EPDs are actually at or below breed average for these traits would be only .014 for weaning weight and .023 for 120-day weight. Even though accuracies are high for animal A for growth traits, accuracies remain low, and prediction errors are correspondingly high, for maternal traits. This result reflects the low heritabilities of these traits and also suggests that this sire may have few daughters with records in NSIP flocks. Still, the maternal milk EPD of .8 for this sire suggests that his daughters do milk well.

EPDs for animal B demonstrate that high accuracies are not necessarily associated with genetic superiority. This ram has the highest accuracy for weaning weight in the breed, but all that means is that we can be relatively confident that he is an inferior sire. Of course, this is valuable information, but we would be wrong to consider using him just because his accuracies are high.

EPDs and prediction errors for ewes (animals D, E, and F) demonstrate how difficult it is to achieve high levels of accuracy in evaluating individual ewes, although our level of confidence in evaluating the older ewe (animal D) is very similar to that attained for a ram lamb (animal C). Thus genetic improvement in the ewe flock relies more on identifying groups of females with high average EPD values or on keeping sets of daughters of high EPD rams. Only a few older ewes will have enough progeny to achieve small prediction errors. Still, EPDs for maternal milk and percent lamb crop for animal D; weaning and 120-day weight for animal E; and weaning weight for animal F exceed or closely approach their prediction errors, suggesting that these individuals can be used to improve these traits with only modest risk.

Table 1. EPDs, accuracies, and prediction errors for selected animals from the 2000 Suffolk national genetic evaluation

Animal

WW EPD 2.8 -0.8 2.4 0.1 2.8 2.0

Accuracy .44 .51 .23 .22 .22 .15

Prediction error 1.3 1.1 1.8 1.8 2.1 2.0

WT120 EPD 5.6 -3.7 3.3 -0.4 6.1 3.7

Accuracy .39 .46 .22 .21 .21 .11

Prediction error 2.8 2.5 3.6 3.6 4.1 4.1

MM EPD .8 -.5 0.4 1.0 1.1 0.9

Accuracy .11 .19 .04 .11 .07 .05

Prediction error 1.1 1.0 1.2 1.1 1.3 1.2

NB EPD -3.4 -1.0 -0.3 10.6 2.7 6.9

Accuracy .22 .04 .08 .20 .12 .10

Prediction error 8.0 9.5 9.0 8.0 10.0 9.0

The main use of the prediction error comes when you are forced to choose between a young ram with very high EPDs but also with relatively high prediction errors and an older progeny-tested ram with good, but lower, EPDs and prediction errors. Again, the issue is risk aversion. If your goal is the leap to an elite position in the breed and if you are willing (and have the resources) to take some chances to get there, the younger ram will be the best choice. Over time, it is always better to go with the higher EPD animals, but when prediction errors are large, you may have to weather a few disappointments along the way. On the other hand, if consistency and reliability of production are key to you, you may pay more attention to prediction error, preferring to use proven rams with less risk of future changes in EPD. But overall genetic progress in the flock may be slower.

A rough guideline to assessing reliability of an EPD is that if the EPD exceeds the possible change value, an animal with a positive EPD is unlikely (one chance in six) to drop below zero in the future, and an animal with a negative EPD is not very likely to move above the average. Also, small differences in possible change are not worth worrying about. The issues of importance come when making choices between ram lambs and progeny-tested sires or between adult ewes and ewe lambs. Differences in prediction errors among ewes in the breeding flock are almost never large enough to be important. Focus on EPDs in selecting and culling breeding ewes. Don’t worry about small differences in possible change!

Finally, realize that without widespread AI, the sheep industry will not have the large numbers of proven sires found in dairy cattle. In most cases, our objective is not to find a few exceptional rams, although when such animals do emerge, they will, of course, be welcome. Our goal is to select groups of replacement ewes and rams that will provide consistent genetic improvement. Thus flocks of reasonable size need to focus on the average genetic merit of the rams purchased or the ewe lambs retained each year.