Previous
Next

Outliers

CO-4: Distinguish among different measurement scales, choose the appropriate descriptive and inferential statistical methods based on these distinctions, and interpret the results.
LO 4.4: Using appropriate graphical displays and/or numerical measures, describe the distribution of a quantitative variable in context: a) describe the overall pattern, b) describe striking deviations from the pattern
LO 4.7: Define and describe the features of the distribution of one quantitative variable (shape, center, spread, outliers).
Video: Outliers (2:30)

Using the IQR to Detect Outliers

LO 4.15: Define and use the 1.5(IQR) and 3(IQR) criterion to identify potential outliers and extreme outliers.

So far we have quantified the idea of center, and we are in the middle of the discussion about measuring spread, but we haven’t really talked about a method or rule that will help us classify extreme observations as outliers. The IQR is commonly used as the basis for a rule of thumb for identifying outliers.

The 1.5(IQR) Criterion for Outliers

An observation is considered a suspected outlier or potential outlier if it is:

  • below Q1 – 1.5(IQR) or
  • above Q3 + 1.5(IQR)

The following picture (not to scale) illustrates this rule:

A line representing all of the data. The data is ordered so that the minimum point is the leftmost on the line and the maximum point is the rightmost. At the center of the line is M, the median, and to the left of M is Q1. Even farther to the left of Q1 is Q1-1.5(IQR). Points farther left than this are suspected outliers. To the right of M is Q3, and farther to the right is Q3+1.5(IQR). Points even farther than this are also suspected outliers.

EXAMPLE: Best Actress Oscar Winners

We will continue with the Best Actress Oscar winners example (Link to the Best Actress Oscar Winners data).

34 34 26 37 42 41 35 31 41 33 30 74 33 49 38 61 21 41 26 80 43 29 33 35 45 49 39 34 26 25 35 33

Recall that when we first looked at the histogram of ages of Best Actress Oscar winners, there were three observations that looked like possible outliers:

A histogram of the Oscar winners in which for x=62, x=74, and x=80, the frequency is 1. Those three points are thought to be possible outliers.

We can now use the 1.5(IQR) criterion to check whether the three highest ages should indeed be classified as potential outliers:

  • For this example, we found Q1 = 32 and Q3 = 41.5 which give an IQR = 9.5
  • Q1 – 1.5 (IQR) = 32 – (1.5)(9.5) = 17.75
  • Q3 + 1.5 (IQR) = 41.5 + (1.5)(9.5) = 55.75

The 1.5(IQR) criterion tells us that any observation with an age that is below 17.75 or above 55.75 is considered a suspected outlier.

We therefore conclude that the observations with ages of 61, 74 and 80 should be flagged as suspected outliers in the distribution of ages. Note that since the smallest observation is 21, there are no suspected low outliers in this distribution.

 

The 3(IQR) Criterion for Outliers

An observation is considered an EXTREME outlier if it is:

  • below Q1 – 3(IQR) or
  • above Q3 + 3(IQR)

EXAMPLE: Best Actress Oscar Winners

We can now use the 3(IQR) criterion to check whether any of the three suspected outliers can be classified as extreme outliers:

  • For this example, we found Q1 = 32 and Q3 = 41.5 which give an IQR = 9.5
  • Q1 – 3 (IQR) = 32 – (3)(9.5) = 3.5
  • Q3 + 3 (IQR) = 41.5 + (3)(9.5) = 70

The 3(IQR) criterion tells us that any observation that is below 3.5 or above 70 is considered an extreme outlier.

We therefore conclude that the observations with ages 74 and 80 should be flagged as extreme outliers in the distribution of ages.

Note that since there were no suspected outliers on the low end there can be no extreme outliers on the low end of the distribution. Thus there was no real need for us to calculate the low cutoff for extreme outliers, i.e. Q1 – 3(IQR) = 3.5.

See the histogram below, and consider the outliers individually.

  • The observation with age 62 is visually much closer to the center of the data. We might have a difficult time deciding if this value is really an outlier using this graph alone.
  • However, the ages of 74 and 80 are clearly far from the bulk of the distribution. We might feel very comfortable deciding these values are outliers based only on the graph.

A histogram of the Oscar winners in which for x=62, x=74, and x=80, the frequency is 1. Those three points are thought to be possible outliers.

 

Understanding Outliers

LO 4.16: Discuss possible methods for handling outliers in practice.

We just practiced one way to ‘flag’ possible outliers. Why is it important to identify possible outliers, and how should they be dealt with? The answers to these questions depend on the reasons for the outlying values. Here are several possibilities:

  1. Even though it is an extreme value, if an outlier can be understood to have been produced by essentially the same sort of physical or biological process as the rest of the data, and if such extreme values are expected to eventually occur again, then such an outlier indicates something important and interesting about the process you’re investigating, and it should be kept in the data.
  2. If an outlier can be explained to have been produced under fundamentally different conditions from the rest of the data (or by a fundamentally different process), such an outlier can be removed from the data if your goal is to investigate only the process that produced the rest of the data.
  3. An outlier might indicate a mistake in the data (like a typo, or a measuring error), in which case it should be corrected if possible or else removed from the data before calculating summary statistics or making inferences from the data (and the reason for the mistake should be investigated).

Here are examples of each of these types of outliers:

  1. The following histogram displays the magnitude of 460 earthquakes in California, occurring in the year 2000, between August 28 and September 9:

    A histogram titled "California Earthquakes, Aug 28,2000 - Sep 9,2000". The histogram is skewed-right. Frequency on the Y-axis ranges from 0 to 90, and on the X-axis is Magnitude in Richter units, from 0 to 5.4 . As we go from left to right across the X-axis, the frequency increases to the mode at x=1.2, y=90, then it decreases to 0 after x=3.6. However, beyond 4.8, we see a small bar representing a frequency of 1.

    Identifying the outlier: On the very far right edge of the display (beyond 4.8), we see a low bar; this represents one earthquake (because the bar has height of 1) that was much more severe than the others in the data.

    Understanding the outlier: In this case, the outlier represents a much stronger earthquake, which is relatively rarer than the smaller quakes that happen more frequently in California.

    How to handle the outlier: For many purposes, the relatively severe quakes represented by the outlier might be the most important (because, for instance, that sort of quake has the potential to do more damage to people and infrastructure). The smaller-magnitude quakes might not do any damage, or even be felt at all. So, for many purposes it could be important to keep this outlier in the data.

  2. The following histogram displays the monthly percent return on the stock of Phillip Morris (a large tobacco company) from July 1990 to May 1997:

    A histogram titled "Phillip Morris Monthly Stock Return, July 1990 - May 1997. On the Y-axis is Frequency, from 0 to 30. On the X-axis is Monthy Stock Return in percent. It ranges from -30 to 20. The histogram is skewed-left. At the very left, between at the interval x=(-30, -25), a bar indicating frequency of 1 appears. Then, we see no bar until x=-15, where there is a bar of frequency 5. As we continue moving right along the x-axis, frequency increases to the mode of 30 at the interval x=(0,5), and then decreases, until reaching a frequency of 5 at the interval x=(15,20).

    Identifying the outlier: On the display, we see a low bar far to the left of the others; this represents one month’s return (because the bar has height of 1), where the value of Phillip Morris stock was unusually low.

    Understanding the outlier: The explanation for this particular outlier is that, in the early 1990s, there were highly-publicized federal hearings being conducted regarding the addictiveness of smoking, and there was growing public sentiment against the tobacco companies. The unusually low monthly value in the Phillip Morris dataset was due to public pressure against smoking, which negatively affected the company’s stock for that particular month.

    How to handle the outlier: In this case, the outlier was due to unusual conditions during one particular month that aren’t expected to be repeated, and that were fundamentally different from the conditions that produced the values in all the other months. So in this case, it would be reasonable to remove the outlier, if we wanted to characterize the “typical” monthly return on Phillip Morris stock.

  3. When archaeologists dig up objects such as pieces of ancient pottery, chemical analysis can be performed on the artifacts. The chemical content of pottery can vary depending on the type of clay as well as the particular manufacturing technique. The following histogram displays the results of one such actual chemical analysis, performed on 48 ancient Roman pottery artifacts from archaeological sites in Britain:

    A histogram titled "Manganous Oxide Content in a sample of Ancient Roman Pottery". The X-axis is labeled "number of pottery shards", and ranges from 0 to 20. The Y-axis is labeled "manganous oxide [MnO] content" and ranges from 0.0 to 0.4 . The histogram is skewed-right. Here are the bars: x=0.0,y=10; x=0.05,y=13; x=0.1,y=18; x=0.15,y=5; x=0.20,y=1; x=0.4,y=1. Note that there are no shards for x=0.25 to x=0.35As appeared in Tubb, et al. (1980). “The analysis of Romano-British pottery by atomic absorption spectrophotometry.” Archaeometry, vol. 22, reprinted in Statistics in Archaeology by Michael Baxter, p. 21.

    Identifying the outlier: On the display, we see a low bar far to the right of the others; this represents one piece of pottery (because the bar has a height of 1), which has a suspiciously high manganous oxide value.

    Understanding the outlier: Based on comparison with other pieces of pottery found at the same site, and based on expert understanding of the typical content of this particular compound, it was concluded that the unusually high value was most likely a typo that was made when the data were published in the original 1980 paper (it was typed as “.394” but it was probably meant to be “.094”).

    How to handle the outlier: In this case, since the outlier was judged to be a mistake, it should be removed from the data before further analysis. In fact, removing the outlier is useful not only because it’s a mistake, but also because doing so reveals important structure that was otherwise hidden. This feature is evident on the next display:

    A histogram titled " Histogram without the outlier" The Y-axis is labeled "number of pottery shards", and it ranges from 0 to 12. The X-axis is labeled "manganous oxide [MnO] content" and ranges from 0.00 to about 0.18. Going from left to right along the X-axis reveals that at x=0, there is a frequency of 10. Then, there are no bars until x=0.4 . From here the bars increase in height until x=0.08, where the frequency is 12. Then the bars begin to decrease.

    When the outlier is removed, the display is re-scaled so that now we can see the set of 10 pottery pieces that had almost no manganous oxide. These 10 pieces might have been made with a different potting technique, so identifying them as different from the rest is historically useful. This feature was only evident after the outlier was removed.

Reading: Outliers (≈ 1400 words)

Tagged as: , , , , , , , , , , , , , , ,

Previous
Next