A common question that we get from clients is “Do I have enough responses for my survey?” The honest answer: it depends. What does it depend on? Actually, a few different things, some related to statistics and some related to business.
If you prefer the “short” version – i.e. Just give me a number please – read the “short version” below. If you prefer the more comprehensive explanation, don’t stop there!
Determining how many responses you need for your survey (Short Version)
Determine the population size: In sports, this can sometimes be easy – the number of Season Ticket Holders, how many people attended last night’s Draft Party, how many people live in your market, etc. Other times, it gets a little trickier – how many people in your market are fans, how many people shop for merchandise, etc. In cases where the exact population size is not known, an educated guess will suffice.
Calculate the ideal sample size: There exists a formula that will calculate the sample size needed to achieve representative results to your survey. Many Internet-based calculators will do this for you. At Turnkey, we prefer one from Raosoft^. The calculator will ask you for margin of error (we recommend 5%), confidence level (95% is the norm), population size (see above) and response distribution (50% is default, leave as is).
- For large populations (over 20k), use 375 as a good estimate of how many responses to shoot for.
- For small populations (under 200), it will be very difficult to get a response rate large enough to get a 5% margin of error. In those instances, we suggesting aiming for a 25% response rate to end up with a reliable data set.
- Remember, when cutting the data into smaller groups (ex: looking at responses by gender or seat location), each of these subgroups will need to have a sufficient number of responses. This generally necessitates a larger number of total responses to ensure enough responses in each subgroup.
Determining how many responses you need for your survey (Additional Thoughts)
In the field of statistics, the following parameters are dependent on each other:
- Sample size
- Level of confidence
- Margin of error
- Statistical power (i.e. the likelihood that your data analysis will correctly lead to a significant conclusion)
Therefore, when a client asks us “How much sample do I need?”, the true answer should be “It depends on your confidence level, margin of error tolerance, and how much power you want.” If you want a high level of confidence, a small margin of error, and a lot of power, you will require more sample than if you’re content with less confidence, a wider margin of error, and/or less power.
Before settling on a target sample size, consider the following questions:
- What size margin of error do I really need? Keep in mind that in business situations, you generally want a sizeable gap in responses before acting. If 60% of fans like option A and 62% like option B, but B is the more costly option, even a 2% margin of error (which would generally be considered very low) doesn’t give you a definitive idea as to which option is actually more popular. In business, generally speaking, 5% either way is acceptable. This is merely a rule of thumb, NOT a definition.
- How confident do I need to be? There are times when you really need to be sure, like super-sure. And there are times when you can be less sure. You might want to be 99.999% sure before changing the brand of hot dogs served at games, but only 90% when deciding to try a new hot dog and soda combination. If you need less confidence, you don’t require as large of a sample.
- What happens if I fail to find a significant difference? The opposite of statistical power is opportunity loss. You obviously want to find a difference if it’s there. But if the opportunity lost is so-so, again you have the flexibility to lower your sample size.
Another key point to consider is that the law of diminishing returns applies to sample size. As sample size grows, the result is a smaller margin of error – but once the sample is past n=400-500, the margin of error doesn’t reduce much, even when the sample size is dramatically increased. In other words, you need a lot of sample to achieve even a slight improvement in margin of error.
Turnkey recommends thinking about these questions when calculating your ideal sample size. If you are getting 1,000 or even 750 responses per subgroup of interest, it’s likely that you are actually oversampling. Oversampling can lead to survey fatigue by respondents and low response rates for future studies. Instead, consider getting fewer responses – and doing more surveys!