A/B testing

A campaign can carry an experiment: two to four variants, each with its own full content map, plus an optional sample rate that holds part of the audience back for the winner.

Defining one

Dashboard: content step → A/B test → name each variant, edit its copy in its own tab, set weights and the sample.

API:

{
  "name": "Cart reminder — subject line test",
  "content": { "en": { "title": "You left something behind", "body": "…" }, "_default": "en" },
  "experiment": {
    "sampleRate": 0.5,
    "variants": [
      { "id": "a", "name": "Plain reminder",  "weight": 1, "content": { "en": { "title": "You left something behind", "body": "…" }, "_default": "en" } },
      { "id": "b", "name": "With item count", "weight": 1, "content": { "en": { "title": "{{cartItems}} items are waiting", "body": "…" }, "_default": "en" } }
    ]
  },
  "segment": { "field": "tags.cartItems", "op": "gte", "value": 1 }
}

Field	Rules
variants[].id	^[a-z0-9_-]{1,8}$, unique
variants[].weight	relative, > 0 (2:1 = two thirds / one third)
variants[].content	a complete content map, same rules as the campaign's
sampleRate	0.01–1. 0.5 = half the audience takes part; the other half is held back
top-level content	fallback / what the summary shows; each variant carries its own

Deterministic assignment

Nothing is random. For each user:

bucket = sha256("<campaignId>|<userId>|sample")  →  in the test if bucket < sampleRate
variant = weighted pick by sha256("<campaignId>|<userId>|variant")

Consequences you can rely on:

• A user with three devices sees the same variant on all three.
• Re-running fan-out (a retry) assigns the same variant again.
• The hold-out is exactly the complement of the sample, by the same key — the two halves can never overlap.

Reading the results

The report's A/B table shows per variant: sent, delivered (+ rate), open rate, click rate. The tracking id embeds the variant (…|platform|variant), so every SDK ping increments both the campaign counter and stats.variants.<id>.<type> without a lookup. Rates use delivered as the denominator, like everywhere else.

Give it time: opens trail deliveries by hours. The hourly chart tells you when the curves have flattened.

Sending the winner

With sampleRate < 1 there is a held-out remainder. Pick a variant:

• Dashboard: Send to the rest on the report.
• API: POST /v1/campaigns/:id/experiment/winner { "variant": "b" } → 202 { id: <new campaign> }.

What happens:

1. A new campaign is created with the winner's content and experiment.remainderOf = <parent>.
2. Its fan-out uses the same hash key with !inSample — exactly the users who were not in the test, no more, no less.
3. The parent records winner and winnerCampaignId; the report links to the follow-up (Winner send's report).
4. A second winner call answers 409.

With sampleRate: 1 everyone was in the test; there is no remainder and no winner button.

Tips

• Test one thing (title, CTA, image). Two changes at once tell you nothing.
• 2 variants × 50 % sample on 10 000 users = 2 500 per arm — enough for open-rate differences of a few points to be real. Below ~500 per arm, treat differences as noise.
• Variants can use different buttons and images, not just text.