"What questions should we ask in a usability test?" comes up constantly. The answer is uncomfortable: the heart of usability testing isn't questions, it's observation.
Designing around questions actively obscures what you should be watching. This post covers task design, the think-aloud protocol, what to actually observe, and where (sparingly) questions fit.
What usability testing is
Usability testing is a qualitative method where you watch real users interact with a product and identify the friction.
| Usability test | User interview | |
|---|---|---|
| Center of gravity | Observed behavior | Conversation |
| Subject of study | The act of using the product | Memories of using it |
| Setting | Hands on the product | Talking only |
| What you find | Specific friction | Motivations, values |
In a usability test, what you watch matters more than what you ask.
Design around tasks, not questions
The unit of design in usability testing is the task.
What a task is
A specific instruction asking the participant to actually do something.
Example task:
"You're a 40-year-old professional. Your team is planning an offsite next month
with 30 attendees. You need to coordinate availability across the team.
Use this site to send out a date-availability survey."
A good task includes:
- A concrete situation (who you are, why you're doing this)
- A clear goal
- Relevant constraints (time, budget, etc.)
"Make a survey" is too abstract. Wrap it in a scenario and people behave more like they would in real life.
Task design principles
Set a goal, not a procedure
× "Click 'New' in the top-left menu" (procedural)
○ "You need to send a 30-person availability survey. Give it a try." (goal only)
The moment you give procedural hints, you stop observing what users would naturally do.
Cap at 3–5 tasks per session
A 60–90 minute session can sustain 3–5 tasks before fatigue takes over.
Ramp difficulty
Task 1: Easy (basic feature confirmation)
Task 2: Medium (some hunting required)
Task 3: Hard (complex flow)
Task 4: Applied (typical real-world scenario)
Starting with the hardest task tanks the rest of the session — you've put the participant in defensive mode.
Failed tasks are valuable data
A task that doesn't get completed is also a finding — it identifies a real usability problem. Your goal isn't a 100% completion rate.
Think-aloud protocol
The core technique. Ask participants to narrate their thoughts as they work.
How it sounds
Moderator: "Could you tell me what you're thinking as you go?"
User: "Okay, I want to make a new survey, so… top right, there should
be a 'New' button somewhere… oh, I see a '+' here. Maybe that's it?
Although I'm not sure if this lets me make a survey or just…"
Surfacing hesitation, exploration, and realization out loud exposes the cognitive process that screen recordings alone can't capture.
Think-aloud rules
- Don't explain. "That's the button you want" defeats the purpose.
- Prompt silent participants. "What are you thinking right now?" — gentle reminders only.
- Don't evaluate. No "right" or "wrong." Stay neutral.
When questions do help (sparingly)
Questions are auxiliary. They go in mid-task and post-task, not as the structure.
Mid-task probes (use sparingly)
1. "What are you thinking right now?" (think-aloud prompt)
2. "What are you looking for?" (clarify intent)
3. "Was that what you expected?" (surface expectation gap)
Post-task questions
1. "What was hardest about that task?"
2. "What felt awkward or confusing?"
3. "Anything that worked differently than you expected?"
4. "If you had to do it again, what would you change?"
5. "If you had to rate that task 1–10, what would you give it?" (light quant)
End-of-session questions
1. "Looking back at the whole session, what stood out the most?"
2. "Anything else you wanted to mention?"
3. "If you were describing this to a colleague, what would you say?"
"Can you do it?" — why this question kills your test
The single most common usability-test mistake is asking, "Can you do this?"
Why it fails
× Moderator: "Do you think you can build a survey?"
User: "Yeah, looks doable."
Reality: 3 minutes in, they're stuck. 5 minutes in, they give up.
When you ask "can you?", participants almost always say yes. Reasons:
- They want to preserve self-image
- They don't want to disappoint the moderator
- Saying "no" makes them feel incompetent
The verbal report and the actual behavior diverge sharply.
What to do instead
○ Moderator: "Go ahead and give it a try. Talk through what you're thinking."
(Watch what actually happens.)
User: "Okay, where do I start…"
(You can now observe the hesitation.)
Have them do the thing. Watch them do it. Don't ask whether they can.
Five things to watch for
1. Hesitation points
A few seconds of "where am I?" almost always indicates a usability problem worth capturing.
2. Wrong clicks
When users click the wrong element, you've found a mismatch between their mental model and your design.
3. Tone and facial expressions
"Huh?" "Wait, what?" "Oh, I see…" — these vocal cues mark the moment of confusion or insight.
4. The moment of giving up
When a user abandons the task — that's the worst friction point in the entire flow. Always log it.
5. Reinterpretation
Moments where the user assigns meaning to something different from the designer's intent. "I thought that was a [X] button" is gold.
Test formats
In-person
- Same physical room as the participant
- Body language and facial cues are observable
- Highest logistical cost
Remote moderated
- Zoom + screen share + recording
- Lower travel and scheduling friction
- Loses some of the body-language signal
Async / unmoderated
- Tools like Maze, UserTesting, Lookback
- Participants on their own time
- High volume, lower observational depth
Combining with surveys
Usability tests run at n = 5–10 typically. That's enough for qualitative pattern-finding but not for quantification. Surveys complement this:
1. Survey identifies users who report difficulty
↓
2. Recruit them into usability tests
↓
3. Find the specific friction points
↓
4. Ship UI changes
↓
5. Re-survey to validate
The qualitative-quantitative pairing is the same dynamic covered in User interviews vs. surveys.
Common usability test mistakes
Mistake 1: Tasks too abstract
× "Try out the site"
○ "You're [scenario], trying to [goal]. Give it a try."
Mistake 2: Helping too much
When users get stuck, the urge to say "click here" is strong. Resist — you're losing the observation in exchange for a meaningless completion.
Mistake 3: Concluding from n=1
A single participant might be an edge case. At least 5 participants is the standard before drawing patterns.
Mistake 4: Testing only with insiders
Friends and coworkers know the product too well. Their reactions don't represent real users. Recruit people unfamiliar with the product.
Mistake 5: Forcing quantification
Numerical "ease ratings" feel comforting but aren't the point of usability testing. Don't reduce qualitative observations to a single number for executive consumption — you're throwing away the actual finding.
How Repoan plugs in
Repoan handles the survey side that wraps around your usability tests:
- Pre-test screening surveys — recruit the right participants
- Post-test debrief surveys — capture immediate impressions in structured form
- Pre/post improvement validation — quantify the impact of UI changes
- Cross-session aggregation — connect individual findings to broader patterns
Summary
- Usability testing is observation; questions are auxiliary
- Task design carries 90% of the work — concrete scenarios, goals not procedures, 3–5 tasks per session
- Think-aloud surfaces the cognition you can't see on screen
- "Can you do it?" is the question that kills your test
- n = 5–10 is enough for pattern-finding; combine with surveys for scale validation
There are insights about your UX that surveys structurally cannot reach. Usability testing reaches them. Building the observation muscle is one of the durable, AI-resistant skills in product work.