Learn by Doing – Statistical vs. Practical Significance

Published: October 22nd, 2013

Category: Activity 1: Learn By Doing

The purpose of this activity is to give you guided practice exploring the effect of sample size on the significance of sample results, and help you get a better sense of this effect. Another important goal of this activity is to help you understand the distinction between statistical significance and practical importance.

Background:

For this activity, we will use example 1. Click here for the associated questions. Here is a summary of what we have found:

A large circle represents the population of products produced by the machine (following the repair). We want to know p about this population, or what is the proportion of defective products. The two hypotheses are H_0: p = .20 and H_a: p < .20. We take a sample of 400 products, represented by a smaller circle. We find that 64 of these are defective. p-hat = 64/400 = .16, and z = -2 and p-value = .023 . Since the p-value is small we conclude that H_0 can be rejected.

The results of this study—64 defective products out of 400—were statistically significant in the sense that they provided enough evidence to conclude that the repair indeed reduced the proportion of defective products from 0.20 (the proportion prior to the repair).

Even though the results—a sample proportion of defective products of 0.16—are statistically significant, it is not clear whether the results indicate that the repair was effective enough to meet the company’s needs, or, in other words, whether these results have a practical importance.

If the company expected the repair to eliminate defective products almost entirely, then even though statistically, the results indicate a significant reduction in the proportion of defective products, this reduction has very little practical importance, because the repair was not effective in achieving what it was supposed to.

To make sure you understand this important distinction between statistical significance and practical importance, we will push this a bit further.

Consider the same example, but suppose that when the company examined the 400 randomly selected products, they found that 78 of them were defective (instead of 64 in the original problem):

A large circle represents the entire population of products produced by the machine (following the repair). We want to find p for this population, which is the proportion of defective products. Our two hypotheses are H_0: p=.20, H_a: p<.20. We create a sample of 400 products (represented by a smaller circle). We find that 78 are defective, so p-hat = 78/400 = .195

Consider now another variation on the same problem. Assume now that over a period of a month following the repair, the company randomly selected 20,000 products, and found that 3,900 of them were defective.

A large circle represents the entire population of products produced by the machine (following the repair). We want to find p for this population, which is the proportion of defective products. Our two hypotheses are H_0: p=.20, H_a: p<.20. We create a sample of 20000 products (represented by a smaller circle). We find that 3900 are defective, so p-hat = 3900/20000 = .195

Note that the sample proportion of defective products is the same as before , 0.195, which as we established before, does not indicate any practically important reduction in the proportion of defective products.

Summary: This is perhaps an “extreme” example, yet it is effective in illustrating the important distinction between practical importance and statistical significance. A reduction of 0.005 (or 0.5%) in the proportion of defective products probably does not carry any practical importance, however, because of the large sample size, this reduction is statistically significant. In general, with a sufficiently large sample size you can make any result that has very little practical importance statistically significant. This suggests that when interpreting the results of a test, you should always think not only about the statistical significance of the results but also about their practical importance.


This document is linked from More about Hypothesis Testing.