When people hear "survey error," they usually picture "not enough samples." In reality, the errors that more sample size can't fix are more dangerous, and they threaten research credibility more fundamentally.
This article organizes the eight major biases and errors contaminating survey data, with counters for each.
Two error types — sampling vs. non-sampling
| Error type | Cause | Counter |
|---|---|---|
| Sampling error | Random variation in drawing samples from the population | Increase sample size |
| Non-sampling error | Systemic shifts from non-sampling factors | Design and operational discipline |
Sampling error is calculable and the counter is simple (more samples). Non-sampling error doesn't go away with more samples — which is what makes it the harder problem.
Eight biases organized
Bias 1: Selection bias
The respondents don't represent the population.
Examples
- Customer satisfaction surveys are answered by strongly satisfied and strongly dissatisfied people, not the middle
- Post-cancellation surveys are answered by the especially angry or the calmly reflective — not the silent middle
- Voluntary e-commerce reviews concentrate at the two extremes
Counters
- Random sampling of distribution targets
- Response incentives to pull the middle in
- Confirm respondent attributes match the distribution audience
- Always footnote results with "selection bias possible"
Bias 2: Response bias
Respondents answer differently from their true beliefs. Several subtypes.
Subtype 1: Social desirability bias
Respondents give the "correct" or "expected" answer.
Q: "How many minutes a day do you exercise?"
Truth: 30 min
Answer: 60 min (closer to the ideal they'd like to project)
Counter
- Emphasize anonymity (named surveys produce more of this)
- Include "easy" options ("Never")
- Cross-check with behavioral data when possible
Subtype 2: Acquiescence bias
Tendency to say "yes." Especially prevalent in some cultures.
Q: "Are you satisfied with our product?"
Answer: tends toward "yes" regardless of true feeling
Counter
- Reverse-keyed items ("What are you dissatisfied with?")
- Use graded scales ("Very satisfied / Satisfied / Neutral / Dissatisfied / Very dissatisfied"), not yes/no
- Prefer 5-point Likert over binary
Subtype 3: Central tendency bias
People pick "3" on a 5-point scale. Avoiding extremes.
Almost everyone picks 3 or 4 → no signal
Counter
- 6 or 7 points (no clear middle, or clearly positioned)
- Intentionally drop "neutral" (4-point scale)
- Anchor questions in concrete behavior ("How likely are you to recommend X to a friend?")
Bias 3: Order bias
Question order changes answers.
Examples
- Asking "what's good?" first then overall satisfaction → satisfaction goes up
- Asking "what's bad?" first → satisfaction goes down
- Similar questions back-to-back → later questions get answered carelessly
Counter
- Place important questions at the beginning or end
- Overall satisfaction first, details after
- Don't run similar questions back-to-back (shuffle order)
Bias 4: Leading question bias
The question itself steers toward an answer.
Examples
✗ "How satisfied are you with our easy-to-use interface?"
("easy-to-use" is presupposed)
○ "Rate the usability of the interface."
✗ "How would you rate our industry-leading support quality?"
○ "Rate the support quality."
Counter
- No adjectives or modifiers in the question
- Always offer a neutral option
- Have a third party review for leading tone
Bias 5: Recall bias
Respondents can't accurately remember past events.
Examples
- "Usage frequency in the past 3 months" gets overestimated (dragged by recent vivid days)
- "Satisfaction last time you used it" gets confused with average satisfaction
Counter
- Ask about recent experience ("Tell me about the last time you used it")
- "How was the last time?" beats "How is it generally?"
- Combine with objective data (logs)
Bias 6: Priming effect
Earlier questions influence later answers.
Examples
Q1: "How do you feel about the current economic situation?"
Q2: "Rate your life satisfaction"
→ Q1 having primed economic anxiety, Q2 satisfaction drops
Counter
- Add buffer questions between sensitive ones
- Don't put answer-influencing questions ahead of important ones
- A/B test order to measure priming impact
Bias 7: Non-response bias
The opinions of people who didn't respond aren't reflected.
Examples
- 20% response rate showing "80% satisfied" → the 80% who didn't respond may be dissatisfied
- Exit surveys don't reach the truly angry (they leave silently)
Counter
- Reminder sends to lift response rate
- Phone or in-person follow-up with a sub-sample of non-respondents
- Compare attributes of respondents vs. non-respondents
- Don't claim "overall trend"; say "trend among respondents"
Bias 8: Survey fatigue bias
Later questions in long surveys get lower-quality answers.
Examples
- In a 30-question survey, questions 25+ get careless answers
- Open text becomes "n/a" repeatedly
- All "3" responses
Counter
- Cut question count to the essentials
- Important questions early
- Show a progress bar
- Exclude responses with abnormally short completion times
"Question design matters more than sample size"
Looking at the eight biases, almost none are solved by more samples:
- Selection bias → 10,000 respondents still biased toward the easy-to-reach
- Social desirability → everyone exaggerates
- Leading bias → question-text problem, sample-size irrelevant
- Order bias → if everyone sees the same order, everyone gets the same distortion
In other words, "question design and distribution design matter vastly more than sample size."
Related reading:
Practical bias / error checklist
Before launching:
Design
- No leading wording or adjectives in question text?
- Graded options instead of yes/no where possible?
- Reverse-keyed items included to validate response patterns?
- Important questions at the beginning or end?
- Question count ≤ 10?
- Progress bar?
- Anonymity guaranteed, or a clear reason for naming?
Distribution
- Audience representative of the target population?
- Subject line / timing / reminders engineered to raise response?
- Plan to compare respondent vs. non-respondent attributes?
Analysis
- Excluded "all 3s" / "all-middle" patterns?
- Excluded abnormally fast completions?
- Flagged respondents with repeated "n/a" open text?
- Footnoted results with "selection bias possible"?
Where Repoan fits
Repoan supports bias-reducing design with AI:
- AI question check — auto-detect leading phrasing and adjectives
- Question order optimization — bias-aware order suggestions
- Auto-suggest reverse-keyed items — embed Likert reversals
- Question count warnings — alert past 10 questions
- Auto-exclude abnormal responses — detect "all-same" answers and speeders
- Anonymity statement templates — auto-insert trust-building language
Summary
Survey error and bias:
- Two types: sampling error (sample-size fix) and non-sampling error (design fix)
- Eight non-sampling biases: selection, social desirability, acquiescence, central tendency, order, leading, recall, priming, non-response, fatigue
- Most biases are prevented in design and distribution, not in analysis
- "Question design matters more than sample size" is the working truth
- Complete elimination is impossible — always footnote "bias possible"
"Numbers came back, so we know the truth" is wrong. "What biases are baked into these numbers — and given that, what should we decide?" That's the discipline of using data well.