"Once you send the survey, you're done" is a common assumption — and a wrong one. Surveys are A/B testable the same way websites and ads are.
This article covers what to test and how to run the program.
Why A/B test a survey
1. Maximize response rate
Same goal, same audience — design choices can move response rate by 2x or more.
2. Improve response quality
How you phrase a question affects how rich the open-text answers come back.
3. Improve downstream CVR
Thank-you pages and individual-meeting CTAs are continuously improvable.
What to test
1. Question order
Variant A: Satisfaction → open text
Variant B: Open text → satisfaction
Order changes answers (order effect). For time-series comparisons, lock the order. For brand-new surveys, finding the right order is worth the test.
2. Option wording
Variant A: "Very satisfied" / "Satisfied" / "Neutral" / "Dissatisfied" / "Very dissatisfied"
Variant B: "Extremely satisfied" / "Somewhat satisfied" / "Neither" / "Somewhat dissatisfied" / "Extremely dissatisfied"
Subtle wording shifts move the distribution.
3. Required vs. optional
Variant A: Contact info required
Variant B: Contact info optional
Required → completion drops, but individual follow-up becomes possible. Optional → completion rises, but lead count from the survey falls.
4. Email subject and body
Open rates are dominated by subject lines.
Variant A: "Survey request"
Variant B: "[1 minute] Your feedback on last week's webinar"
5. Thank-you page CTA
Variant A: "Book a meeting"
Variant B: "Download the report"
Which downstream action gets more click-through?
6. Incentives
Variant A: $1 Amazon gift card for everyone
Variant B: $50 to 5 randomly drawn winners
Which raises response rate? Depends on industry and audience.
A/B test design
Sample size
Detecting statistically meaningful differences requires roughly 200+ per variant as a floor:
Big effect (CVR lift 1.5x+): 200 per variant
Medium effect (1.2–1.5x): 500 per variant
Small effect (1.05–1.2x): 1,000+ per variant
Too few samples = even real effects go undetected.
Test duration
Minimum 2 weeks, to absorb day-of-week and time-of-day variation.
Number of variants
Stick to 2 (A/B). With 3+ variants, required sample size explodes.
Isolate one variable
If you change multiple things at once, you can't tell what produced the lift. One change per test is the rule.
Running A/B tests in Repoan
Method 1: Two separate forms
1. Ask the AI to "build variant A's question structure" → form 1
2. Repeat for variant B → form 2
3. Distribute 50/50 randomly
4. Compare aggregates
The simplest method.
Method 2: Same form, different distribution
- Split mailing list in half
- Group A gets subject A
- Group B gets subject B
- Compare open + response rates
Best for email-copy testing.
Method 3: Thank-you page test
1. Form → thank-you page A
2. Form → thank-you page B
3. Compare CTA click rate / final CV rate
Pairs with Repoan's thank-you-LP feature.
Metrics
Primary
Pick one most-important metric:
- Completion rate
- Meeting-booking rate
- Download rate
Secondary
Lower-priority diagnostics:
- Per-question drop-off rate
- Average completion time
- Open-text average length
Guard metrics
"Must not go down" metrics:
- For a "lift meeting booking" test, completion rate must not drop substantially
Statistical significance
"A is 10%, B is 12%" by itself isn't enough. You need statistical significance.
Quick check (chi-square)
Excel CHISQ.TEST handles it:
Variant A: 200 sent, 20 converted → CVR 10%
Variant B: 200 sent, 30 converted → CVR 15%
p = 0.04 (< 0.05, significant)
→ B effect confirmed
Underpowered
A/B at 50 each rarely detects meaningful differences. At minimum 200 each.
What not to do
❌ Too short
3 days, then deciding "B won" — too early. Day-of-week variance pollutes the result.
❌ Non-randomized distribution
"A in the morning, B at night" → time-of-day bias. Fully random distribution.
❌ Changing multiple variables
"Different subject, different body, different CTA" → you don't know what worked.
❌ p-hacking
Repeatedly retesting until "finally significant" is statistical malpractice.
Limits of A/B testing
Bias never fully disappears
Respondent demographics, seasonality, current-events context — A/B testing doesn't absorb everything.
Marginal improvements take time
Moving CVR from 10% to 12% requires a lot of samples. Weigh the cost.
Sometimes you need bold change
Tweaking the existing structure has a ceiling. Sometimes you need to rethink the framework.
Summary
A/B testing surveys, the essentials:
- Isolate one variable per test
- 200+ samples per variant
- 2+ weeks of runtime
- Check statistical significance
- Distinguish primary from guard metrics
In Repoan, A/B testing works via creating multiple forms and comparing. AI reports let you compare the aggregates side by side, efficiently.