Analysis for people running surveys.
Most survey findings that fall apart do so because of who answered, not because of what they said.
- Read6 min
- Sections4
Why do my survey results change when I include partial responses?
Because the people who abandon a survey are not a random sample of the people who start one. Dropout usually correlates with the thing being measured, so excluding partial responses biases every comparison in the same direction. Check completion by segment before reporting any difference between groups; if dropout tracks the answer, the headline is measuring dropout.
01
Completion bias beats sample size
A survey with two thousand responses and a 40% completion rate among unhappy respondents is less trustworthy than one with four hundred responses and even completion. Size does not fix a systematic absence.
The check is straightforward: compare who finished against who started, split by the thing you are measuring. If dropout correlates with the answer, your headline comparison is measuring dropout.
In the worked example, the tenure gap the survey was run to test disappeared entirely once partial responses were included.
02
Reverse-coded questions quietly break the average
Negatively worded items have to be recoded before anything is averaged. That much is standard — but it is worth checking afterwards whether they still behave, because respondents frequently answer them as if they were worded positively.
An item that correlates oddly with the rest of the scale after recoding is not measuring what its neighbours measure, and including it drags the overall score toward noise.
03
Small segments should be counts, not percentages
Below about thirty responses, one person moves a rate by more than the effect being measured. Reporting "Enterprise satisfaction fell 8 points" when Enterprise is twenty-four people is reporting one person changing their mind.
Counts are less satisfying and more honest, and they make the sample size impossible to overlook.
04
Free text is evidence, not decoration
Written answers are usually summarised separately from the scores, which wastes both. A theme appearing in forty comments means something different depending on whether those forty people rated you two or five.
Grouping the text by theme and showing the mean score for each theme is the single most useful thing you can do with it.
See it on a real project
The apparent 0.42-point gap fell to 0.07 once partial responses were included.
Is the difference between customer groups real?Related
Try it on your own file.
Every check described here runs automatically, and your first findings arrive in full before you pay for any of it.
No account needed to start. You only pay when you like what you see.