The four decisions that make a pilot measurable — as a worksheet you complete with the vendor and attach to the agreement. Any section left blank is where the post-hoc storytelling will enter.
Section 1 · The split
- Treatment group. Properties or sections where the product will be enabled.
- Held-back group. Comparable properties or sections that stay unchanged — the counterfactual.
- Selection rule. Why these — ideally “entire eligible set” or a stated criterion, never chosen after results.
Comparability check: the two groups’ weekly trends must move together during the baseline, or the comparison is repaired before the post-period counts.
Section 2 · Metric freeze
- Primary metric and exact definition. e.g. multi-page session share = sessions with >1 pageview ÷ sessions; distinct-article counting stated.
- Secondary metrics.
- Counting rules. Bot filter, de-duplication, distinct-content units.
- Freeze date. Before enablement, in writing.
Section 3 · Baseline
- Instrumentation live from (day zero).
- Measurement-only period (weeks). Floor: 4.
- Enablement date — the intervention anchor. A named day. Every later claim hangs on it.
- Instrumentation faults found and fixed during baseline.
Section 4 · Windows and integrity bounds
- Post-period length. Minimum: one full quarter past launch — long enough to see past the launch bump.
- Launch window vs steady-state window. Both will be reported.
- Week-invalidation rule. Tracking gap, migration or retag on either group ⇒ the week is dropped.
Panel balance: every property present every counted week, or the week is dropped. Analysis ends at the last clean date for the whole panel — not the calendar.
Section 5 · Decision gates
- Success condition (pre-agreed). Treatment exceeds held-back group by ___ points on the primary metric over the steady-state window.
- Failure condition — the vendor completes this sentence. “If ___ by ___, it isn’t working.”
- Off-test (optional, powerful). A scheduled pause late in the pilot; expected: metric returns toward baseline, recovers on resume.
- Decision date and owners.
