A/B Testing: Comparing Product Experiences Fairly

When two product experiences are compared among different self-selected users, differences may come from the users rather than the experience.

An A/B test uses random assignment to make the groups comparable on average.

Randomization creates a fair comparison; disciplined design and measurement make the result useful.

Experiments do not replace judgment about ethics, long-term effects, or practical significance.

A concrete example: onboarding

Eligible new users are assigned:

• A: current onboarding,
• B: shorter onboarding.

The team measures:

• activation,
• 30-day retention,
• support contacts,
• accessibility failures,
• and errors.

Assignment remains stable for each user through the experiment.

Hypothesis

State a prediction before running:

Shorter onboarding will increase activation without increasing support contacts or reducing retention.

This identifies:

• treatment,
• primary outcome,
• guardrails,
• target population.

Eligibility

Define which units can enter:

• new users,
• one country,
• supported app version,
• no prior exposure.

Changing eligibility after seeing results can bias the comparison.

Unit of randomization

Randomize by:

• user,
• account,
• household,
• device,
• session,
• geographic cluster.

Use the unit where interference and repeated exposure are controlled.

Randomizing individual employees inside one collaborating company may contaminate both variants.

Stable assignment

The same unit should see the same variant.

Use a deterministic hash or persisted assignment. Randomizing every page load creates mixed experiences and invalid attribution.

Track assignment independently of whether the user completes the action.

Exposure logging

Assignment does not always mean the user actually saw the treatment.

Log exposure when the variant becomes capable of influencing behavior, while preserving original assignment for intention-to-treat analysis. Missing or variant-specific exposure logging can manufacture an apparent effect.

Ramp-up

Begin with a small percentage to verify assignment, performance, errors, and guardrails before full experimental traffic.

Ramp-up protects users from implementation defects, but changing the population over time should be recorded and considered in analysis.

Control and treatment

The control represents the current or baseline experience.

Treatment changes one defined factor where possible. Bundling redesign, pricing, and notification together makes it difficult to learn which caused the effect.

Some product decisions intentionally test a package, but interpretation should match.

Primary metric

Choose one or a small number of primary outcomes before analysis.

Examples:

• activation,
• purchase completion,
• successful task,
• retained user.

Searching dozens of metrics increases the chance of finding a random "winner."

Guardrail metrics

Guardrails detect unacceptable harm:

• error rate,
• latency,
• refunds,
• complaints,
• accessibility,
• unsubscribe,
• fraud.

A treatment should not win on clicks by breaking trust or reliability.

Sample size

Required sample depends on:

• baseline rate,
• minimum detectable effect,
• desired power,
• significance threshold,
• and variance.

Smaller effects require larger samples.

Choose the smallest effect worth acting on before seeing data.

Statistical significance

A p-value under a chosen model measures how surprising the observed difference would be if there were no true effect.

It does not state:

• probability the treatment is good,
• size of benefit,
• or practical importance.

Report effect estimate and uncertainty interval.

Confidence intervals

An interval communicates a range of effect sizes compatible with data under assumptions.

It helps answer whether the result could be:

• meaningfully positive,
• negligible,
• or harmful.

One binary significant/not-significant label hides this information.

Power

Power is the chance of detecting an effect of the planned size when it exists.

An underpowered experiment often produces inconclusive noise.

Do not interpret "not significant" as proof of no effect.

Duration

Run long enough to include:

• weekly patterns,
• delayed outcomes,
• novelty,
• and enough sample.

Stopping after one unusually strong day can mislead.

The plan should set duration or sequential rule.

Peeking

Repeatedly checking ordinary fixed-horizon significance and stopping when positive inflates false positives.

Use:

• predetermined end,
• or valid sequential testing methods.

Operational safety monitoring can still stop harmful experiments.

Novelty and learning

Users may react differently because a feature is new.

Some treatments need time for:

• learning,
• habit,
• network effects,
• or fatigue.

Measure appropriate long-term outcomes.

Intention-to-treat

Analyze users according to assigned variant, even if they do not fully engage.

This preserves randomization and estimates the effect of offering the experience.

Analyzing only users who completed treatment reintroduces selection bias.

Sample-ratio mismatch

If a 50/50 test receives 60/40 enrollment, something may be wrong:

• assignment bug,
• logging loss,
• eligibility difference,
• variant crash.

Check allocation before trusting outcome analysis.

Interference

One user's treatment can affect another in:

• social networks,
• marketplaces,
• shared accounts,
• collaboration.

Cluster randomization or market-level experiments may be necessary.

Multiple testing

Many variants, metrics, and subgroups increase false discoveries.

Pre-register priorities, adjust inference, and treat exploratory findings as hypotheses for future confirmation.

Do not mine until one favorable slice appears.

Practical significance

A tiny statistically detectable improvement may not justify:

• complexity,
• maintenance,
• user disruption,
• or ethical cost.

Compare expected value and confidence with implementation and long-term effects.

Experiment ethics

Do not randomize away essential safety or informed consent.

Review:

• possible harm,
• fairness,
• privacy,
• vulnerable groups,
• and reversibility.

Some questions should be answered through safer methods.

Experiment platform

A platform manages:

• assignment,
• exposure logging,
• metrics,
• guardrails,
• analysis,
• and audit.

Validate the platform with known A/A tests and sample-ratio checks.

Knowledge check

1. What does random assignment accomplish?
2. Why must assignment remain stable?
3. How do primary and guardrail metrics differ?
4. Why does ordinary repeated peeking increase false positives?
5. What does intention-to-treat preserve?

The one idea to remember

A/B testing fairly compares experiences by randomly and stably assigning eligible units, measuring predefined outcomes and guardrails, and analyzing effect size with uncertainty. Sample planning, duration, interference, ethics, and resistance to selective interpretation determine whether the result deserves action.