PromptSharp › Prompt Library › Product Management › Experiment design: a test that can actually be wrong

Experiment design: a test that can actually be wrong

You want to run an A/B test but aren't sure it'll teach you anything. Design it so a null result is as informative as a win.

The prompt — copy and run it

You are an experimentation lead designing a valid product experiment — a design aid, not a substitute for a statistician on high-stakes calls.

Produce:

A) HYPOTHESIS — a falsifiable statement: changing X will move metric Y by roughly Z, and the mechanism you expect.

B) DESIGN — the primary metric, guardrail metrics, unit of randomization, and what would make this test invalid (contamination, seasonality, too-small population).

C) READOUT PLAN — what result means ship, what means kill, and what means inconclusive — decided BEFORE the data, so the result isn't rationalized after.

D) POWER REALITY — a rough read on whether the expected effect and your traffic can even produce a conclusive result, or if this test is underpowered.

Inputs: [THE CHANGE] · [METRIC + CURRENT BASELINE] · [EXPECTED EFFECT SIZE] · [TRAFFIC / SAMPLE AVAILABLE]

Rules: Do not invent baselines, effect sizes, or significance — use my numbers and flag when the test is likely underpowered. Pre-commit the decision rule. Keep confidential metrics out of consumer AI tools. This designs the test; the statistical call on close ones stays yours. Verify anything uncertain against the source before relying on it.

Why this prompt works

Most product experiments teach nothing because the decision rule gets rationalized after the data and the test was underpowered from the start; pre-committing ship/kill/inconclusive thresholds and forcing a power reality-check turns an A/B into a real decision instrument — and refusing to invent effect sizes keeps an underpowered test from being sold as a clean win.

Frequently asked

Related Product Management prompts