Build a SaaS homepage experiment from buyer intent

Before redesigning your SaaS homepage, write an experiment brief that connects a buyer’s unanswered question to a specific change and a measurable outcome. Define who qualifies for the test and what evidence would justify keeping the change.

Suppose a visitor arrives after comparing support tools. Your headline promises better productivity, but they need to know whether the product works with their help desk. That compatibility question gives you a concrete starting point for a homepage experiment.

The brief below also helps you decide whether you have enough traffic for an A/B test or should investigate the question through usability sessions first.

Start with the question the buyer cannot answer

Choose a decision someone needs to make on the homepage. A first-time visitor might need to understand the product category. Someone comparing vendors might need to confirm an integration. A returning evaluator might want to understand the commitment involved in a demo.

Collect evidence from conversations, observed tasks, or feedback from people who match your intended audience. Record the context: a prospect asking about setup before a demo has a different problem from an existing customer asking for installation help.

Keep the observation separate from your explanation. If a prospect asks whether you support their help desk, the homepage may make compatibility hard to establish. They may also have found the integration information and need reassurance about a particular configuration. Your research should help distinguish those explanations.

A visit ending without a signup cannot tell you why the visitor left. If you have only aggregate analytics, use the guide to finding where SaaS website visitors hesitate before choosing a change.

Check that the product can answer the question, too. If buyers need an integration you lack, homepage copy cannot solve that gap.

Connect one change to an observable outcome

Use this hypothesis structure:

For [eligible visitors], we believe [observed uncertainty] prevents [next useful action]. Changing [specific page element] should help because [reason]. We will measure [primary outcome] within [conversion window], with [guardrail limits].

Compare “a clearer homepage will convert better” with “naming the supported help desk beside the headline will help arriving evaluators establish compatibility.” The second gives you an explanation to investigate.

Microsoft recommends simple hypotheses with effects that the selected metrics can evaluate. It also recommends separating complex changes when you need to measure each component’s contribution. Microsoft Research: pre-experiment guidance

For a small team, one coherent change is a practical starting point. Revise the integration sentence while keeping the offer and signup flow stable. If you test a complete redesign, interpret the result as evidence about the package; it reveals less about which part helped.

Define eligibility before visitors see the variation

An arrival from a comparison article provides context, but it does not prove readiness to buy. Translate your audience idea into a rule you can apply before showing either version. For example, include visitors first observed through a specified comparison campaign and exclude known employees and existing customers.

Run both versions concurrently and randomly assign eligible visitors. Document how you identify visitors and preserve assignments across repeat visits. Microsoft notes that cookie-based identifiers can change when people delete cookies; your brief should acknowledge such limits. Microsoft Research: choosing a randomization unit

Avoid defining the audience afterward as “people who clicked the integration section.” The revised section could affect who clicks, leaving you with different populations to compare.

Narrow eligibility also reduces available traffic. If the selected audience is too small, investigate the question through research or choose a hypothesis that applies to a broader audience. Changing eligibility just to finish sooner changes the question you are testing.

Measure buyer progress and check measurement quality

Choose one primary outcome close to the buyer decision. For a self-serve product, that might be creating a workspace and completing a meaningful setup action. For a sales-led product, it might be a qualified demo request.

Write the calculation explicitly. For the integration example:

Eligible assigned visitors who create a workspace and connect a supported help desk within seven days of assignment ÷ all eligible assigned visitors in that version.

Count each visitor once under your documented identity rule. Define qualification and event rules before examining results, and give the last enrolled visitors their full conversion window.

Button clicks can help explain behavior, but they may rise without more useful signups. Choose guardrails for outcomes you do not want to damage, such as form failures, page load time, or signups from people unable to use the product. Specify the unacceptable deterioration for each before launch.

Check data quality separately. Do assignment records connect to downstream events in both versions? Do group sizes match the planned allocation within expected random variation? Microsoft treats sample ratio mismatches as reasons to investigate whether an experiment’s results can be trusted. Microsoft Research: monitoring experiments

Set the decision rule and check whether the test is feasible

Estimate the baseline using the same audience and outcome definition. Choose the smallest improvement worth the cost of maintaining the change. Moving from 4% to 5%, for example, is one percentage point or a 25% relative increase.

Then calculate sample requirements using the statistical method you intend to use. Record the baseline, minimum detectable effect, significance threshold, power assumptions, and allocation. The minimum detectable effect is a planning input, not a promised result. Smaller detectable effects generally require larger samples. Optimizely: minimum detectable effect and experiment effort

Check whether the calculator reports visitors per version or a total. Divide the total requirement by eligible weekly visitors allocated to the experiment to estimate enrollment time, then allow for the final visitors’ conversion window.

Choose the analysis method before launch. Fixed-horizon testing uses a predetermined sample and analysis plan. Sequential testing follows different stopping rules. Optimizely documents these as separate configurations; follow the rules of the method you use. Optimizely: fixed-horizon test configuration

Write down how you will handle the possible outcomes:

• Adopt the variation when the planned analysis supports an improvement worth maintaining and guardrails meet their limits.
• Retain the current page when the variation causes unacceptable harm or the evidence rules out an improvement large enough to matter.
• Record an inconclusive result when uncertainty still spans materially different decisions.
• Investigate or invalidate the result when assignment or measurement fails.

Replace “worth maintaining” with your actual threshold and specify how your analysis will assess it. Set an operational deadline as well. If you reach it without enough evidence, record the uncertainty; the higher observed conversion rate alone does not establish a winner.

Worked example: compatibility for an AI support assistant

This hypothetical example concerns an AI assistant that drafts replies inside a supported help desk. Suppose potential buyers repeatedly look for compatibility information in observed sessions, while the homepage mentions the integration only near the bottom.

The team’s hypothesis is:

Among visitors first observed through our help-desk comparison campaign, naming the supported integration beside the headline will increase the share who create a workspace and connect that help desk within seven days of assignment, because they can establish compatibility before starting.

The brief translates that hypothesis into these choices:

The observations and numbers are illustrative, not reported findings or a forecast. This brief still needs an assignment method, guardrail limits, sample requirement, analysis rule, owner, and deadline before launch.

The team next checks the required sample against eligible traffic. If enrollment would take longer than it can keep the offer and campaign reasonably stable, it postpones the conversion experiment.

It can still investigate the explanation. Recruit relevant evaluators and ask them to determine whether the product fits their current help desk, then explain what they would do next. Do not point them toward the integration sentence. GOV.UK recommends clear, neutral task instructions and mostly watching and listening during moderated usability sessions. GOV.UK: moderated usability testing

If participants understand compatibility but hesitate over data access, that concern becomes a candidate for the next hypothesis. If revised copy resolves confusion, the team may adopt it as a research-informed editorial decision. Those sessions do not establish conversion lift.

Copy this homepage experiment brief

Fill in these fields before opening a design tool:

Buyer decision:
Evidence and its context:
Proposed explanation:
Alternative explanation:
Eligibility and exclusions, determined before exposure:
Control and proposed change:
Randomization unit and repeat-visit handling:
Primary outcome numerator and denominator:
Conversion window, starting from:
Guardrails and unacceptable deterioration:
Assignment and measurement checks:
Baseline and smallest worthwhile improvement:
Analysis method and sample-size assumptions:
Required sample, per version and total:
Eligible weekly traffic allocated to the test:
Estimated enrollment time and final measurement date:
Rules for adopt / retain / inconclusive / invalid:
Implementation owner and decision owner:
Operational deadline:

Your next task: document one unanswered buyer question

Complete the buyer decision, evidence, and alternative explanation fields for one homepage change you are considering. If you cannot cite a relevant conversation or observation, arrange a few before redesigning. Once the question is grounded, finish the brief and decide whether to run a conversion experiment or investigate comprehension first.