Usability & A/B Testing

Once you have a prototype or MVP, you must observe it in the wild. This involves qualitative usability tests and quantitative A/B experiments.

Qualitative Usability Testing

When running a usability test, your goal is to observe where users get stuck. Follow these core practices:

• Give tasks, not instructions: Say: "Show me how you would send an invoice to a client named Sarah," not: "Click the blue button, then type Sarah."
• Encourage thinking aloud: Ask the user to talk through what they are seeing and thinking.
• Do not correct them: If they make a mistake, watch how they recover. If you help them, you ruin the test data.

Quantitative Validation & A/B Testing

If you have high traffic, you can run an A/B test. This splits traffic between a Control (current version) and a Variant (new design) to see which performs better on a key metric (e.g., conversion rate).

Before running any validation test, define your success criteria in writing:

"We believe that [solution] will result in [metric change]. We will consider this validated if [threshold] is met."

Example: "We believe a pre-filled template will increase invoice completion. We will consider this validated if the template variant has a completion rate of at least 75% with statistical significance."