When analyzing data, it is common to consider differences – differences between groups, changes over time, etc. Often the first question when looking at the data is, “which differences are significant?” Most people want to know right away if anything is “statistically significant,” but we should also consider whether these differences will have practical implications. Statistical significance shows that there is a high probability that the data reflect true differences in the population surveyed. In market research we not only want to know that differences exist, but we want to know if the differences are large enough to affect our marketing strategies and decisions. This sense of “practical significance” can be more useful than simply focusing on which differences are statistically significant.
Because it is directly tied to sample size, statistical significance can distract from the most important findings. With large sample sizes (over 1,000), even very small differences can become statistically significant. However it is usually not practical to focus on differences of less than 5%. For example, a survey of 1,500 game attendees could show a fall in satisfaction with concessions from 65% to 62% over the course of the season. While this change might be statistically significant, it probably doesn’t make sense to make an effort to address such a small change, especially if there are other areas that show much bigger drop-offs.
Focusing on statistical significance when your sample size is small could be an issue as well. When surveys get a low number of respondents (less than 100), differences will have to be much bigger in order to be statistically significant. While you should be careful drawing conclusions from insignificant data, large differences should not be ignored. If a survey of 40 club seat holders showed a 15% drop in satisfaction with parking, there might be reason to be concerned. Using other data, either open-ended responses or other questions, will help determine whether there is a problem. If other parking related questions show somewhat large gaps as well, then there likely is an issue that should be addressed.
When beginning to analyze data, before focusing on statistical significance, it is a good idea to consider what size differences will have “practical significance” for your organization. That is, figure out how large differences should be to require attention and to affect future decisions. If 3% more upper level seat holders were satisfied with concessions than those in the lower level, would you be worried about the upper level? Or would it have to be a 15% gap before you start looking into the food quality? With such guidelines in mind, you can identify the differences that are truly important and make more actionable conclusions from the data.