Rating and ranking questions are often confused by those utilizing survey research. Rating questions ask respondents to rate a series of elements using a similar scale (ex. “How important are the following service aspects using a 1-5 scale, with 5 being ’extremely important‘ and 1 being ’not at all important’?”). In these instances, the same rating can be used to describe multiple elements. On the other hand, ranking questions ask respondents to rank a series of aspects against one another (ex. “Please rank the following service aspects with 1 being ’most important’ and 5 being ’least important‘). In a ranking question, each “rank” can only be used once. Each question type has its own advantages and disadvantages.
- Easy to understand
- Allow the respondent to select the same value for different elements if they are equally important, rate highly, etc.
- Respondents often rate items similarly, rather than providing a wide range of ratings, and cluster the majority of ratings on the positive/higher end of the scale
- Respondents react to scales in a variety of ways (one particular respondent may opt to never utilize the highest rating on the scale, etc.)
- Matrix rating questions are somewhat long and tiresome to complete, which may drive the respondent to take shortcuts when answering the questions
- Each element receives a unique ranking (i.e., respondents cannot assign the same value to each element)
- Question technique forces discrimination between choices, which provides more statistical “power” (thus allowing analysts to more easily discern real differences where they exist)
- Ranking questions force respondents to choose between two items they may wish to rank equally (e.g. “cost” and “seat location” may be the two most important renewal drivers for a season ticket holder, yet he/she is forced to choose one over the other in a ranking question)
- Response options can be confusing for respondents if rating questions previously asked in the survey use scales with “1” corresponding with the most “negative” response option (i.e., if a rating question offering responses on a 1-5 scale with 5 being ‘extremely important’ and 1 being ‘not at all important’, and then later a ranking question offers responses ranging from ‘1 – Most important’ to ‘5 – Least important’ )
- It typically takes longer to answer ranking questions than rating questions, often because respondents need to compare items against one another
When is it best to use each question type?
Follow Turnkey’s guiding research principle and “let your objectives drive your methodology.” When determining whether to use a rating question or a ranking question, ask yourself if the elements you are asking about should have the ability to receive the same score or if they need to be differentiated between.
For example, when asking a respondent how important a set of benefits are to renewing his/her season ticket package, a rating scale is more appropriate since a few of the benefits may be extremely important for renewing for a particular respondent. On the other hand, when asking respondents which elements should be addressed first when renovations begin in their home stadiums, a ranking question is more applicable since renovation elements will need to be prioritized.
Citation: the content of this article is based in part on “Voice of Vovici Blog: Ranking Questions vs. Rating Questions” (http://blog.vovici.com/blog/bid/18228/Ranking-Questions-vs-Rating-Questions)