I was recently asked whether or not there is a difference or change in response to a survey question if the answer options are arranged in vertical or horizontal format.
The question pertained to Decipher’s responsive survey design, where an 11 point horizontal scale renders in horizontal format on a PC device, but in vertical format on a smartphone device. This change to vertical format occurs when an 11 point horizontal scale is too wide to properly fit the narrow screen of a smartphone. There is an advantage to arranging it this way. Research has shown that mobile-friendly designs improve participation rates by as much as 50% for smartphone users.
Figure 1. PC view – 11 point NPS question, horizontal scale
Impact on Data Quality
Figure 2. A responsive design detects
that an 11 pt horizontal scale is too
wide for the narrow screen width of a
smartphone. So it renders the scale in
vertical format.
Back to the original question. Does the change in the scale arrangement affect the data? In other words: If you take the same rating scale question, but show it in horizontal alignment to a PC user and vertical alignment to a smartphone user – will that cause a difference in the data? Are the two comparable and can you aggregate data from PC users and smartphone users together?
Evidence says that yes, you can.
In a split cell test, we investigated 5, 7 and 11 point scales. We found no significant difference in responses between PC users rating horizontally arranged ratings scales and smartphone users rating vertically arranged rating scales. We even looked at PC data from a 5 column x 9 row grid and compared that to breaking out the nine row attributes into individual vertical scaled questions on the smartphone. We did not see a significant difference there, either.
The findings support that there shouldn’t be a concern when utilizing a responsive design in your survey. It has helped improve participation rates for smartphone users, and we have yet to see evidence that it substantively alters respondent answers to our survey questions.
Figure 3. Frequency response distribution – 11 point scale
The post Horizontal vs. Vertical Rating Scales: Does it Make a Data Difference? appeared first on FocusVision.