A/B Testing an Engagement Feature

An engagement feature is unusually easy to fool yourself about. It ships, the numbers move, and there is always a story available for why. The discipline below is not statistical sophistication, it is mostly a set of decisions made before the data exists.

Randomize at the user#

Assign at the level of the thing you care about retaining. That is a user, keyed on a stable account identifier, not a session and not a device install. Session-level assignment lets the same person see both variants and destroys the comparison. Device-level assignment splits multi-device users across arms and drags the estimate toward zero.

Two consequences worth being deliberate about. Assignment must be sticky across reinstalls, which means it lives server-side against the account, not in local storage. And the assignment has to happen at a point every user in the cohort passes through, otherwise the arms are not comparable — bucketing at "opened the feature" is not randomization, it is self-selection wearing a lab coat.

Pre-register the metric, the duration, and the decision#

Before exposure starts, write down and circulate:

• The primary metric, exactly one, defined operationally (which event, which window, which day boundary).
• The guardrails you would stop for.
• The duration, decided in advance.
• The decision rule — what result ships the feature, what result kills it, and who decides.

The point is not ceremony. It is that a metric chosen after seeing the data is not evidence, and with enough metrics on a dashboard something always moved. Pre-registration is what converts a dashboard into an experiment.

Keep a holdout that outlives the test#

When the test ends, keep a small randomized slice of users permanently off the feature. This costs you a little and buys you the ability to answer questions the experiment cannot: whether the effect persisted, whether it decayed, whether it survived contact with the three other features shipped since. Without a long-run holdout, "our engagement is up" is a statement about the calendar.

Novelty decay: never read the result in week one#

New things get used because they are new. A streak counter, a leaderboard, a camera coaching mode, a redesigned home screen — all of them generate a burst of exploration that has nothing to do with whether the feature has changed anyone's habits.

Our judgement, and the reason we say four weeks and not three days: fitness behavior operates on a weekly rhythm, so anything shorter than four weekly cycles cannot distinguish a habit from a look. If a feature's advantage is gone by week four, it was an interaction, not a mechanic. Plan the duration up front, and resist reading the week-one chart as anything other than "the feature was discovered."

Guardrails, measured with the same seriousness as the target#

An engagement feature can move its target metric while damaging the product. The guardrails that matter most for fitness apps, in our experience:

A win on the target with a guardrail breach is not a win, and the pre-registration should say so before anyone is emotionally invested.

Why peeking inflates false positives#

Checking the dashboard daily and stopping when the p-value dips is not a slightly aggressive version of the experiment. It is a different procedure with different properties. Every additional look is another chance for random fluctuation to cross the threshold, and repeated testing against the same threshold makes a spurious "significant" result substantially more likely than the nominal rate suggests. We are deliberately not quoting a figure for how much more likely — that depends on how often you look and how you stop.

Two ways out, both fine: fix the duration in advance and look once at the end, or adopt a sequential method that is designed for continuous monitoring and spends its error budget accordingly. What is not fine is the informal version, where the team watches daily and calls it when it looks good.

The rollout that proves nothing#

Here is the most common way a team "proves" an engagement feature worked. Ship it to everyone via a staged store rollout, watch the weekly numbers, see them rise, declare victory.

Nothing in that sequence is evidence. The comparison is this week against last week, so the release is confounded with seasonality, marketing, App Store featuring, a competitor's outage, and the weather. Staged rollout slices are a deployment safety mechanism; the users in the early slice are not randomized against a holdout and are typically the fastest updaters, who are also your most engaged users. Reading a store rollout as an experiment measures your update-adoption curve.

What actually determines your sample size#

We are not going to print a formula with invented inputs. Qualitatively, you need more users when the base rate of the outcome is low, when the effect you care about is small, when the metric is high-variance (per-user workout counts are far noisier than a binary "completed a workout this week"), when only part of the audience is ever exposed, and when you want to slice the result by platform or cohort afterward. You need fewer when the metric is binary and common, and when the effect is one you would only bother shipping if it were large.

Work out those inputs first. If the honest conclusion is that your traffic cannot resolve the effect you care about in a reasonable time, that is a finding: pick a bigger swing, pick a metric closer to the change, or accept that you are shipping this one on judgement and say so out loud.