A/B experiments

Bind variant assignments to sessions, then every conversion, funnel step, replay, and log in that session joins the right variant at query time. Sumidata is the analytics side of A/B - your feature-flag service still decides who sees what.

01Mental model

Sumidata is a tracker, not a flag service. You tell it which variant you chose; it joins that to everything else.

An experiment assignment in Sumidata is a tiny row that says: "In session X, experiment Y was running, and the user saw variant Z." The assignment lives in a dedicated table (session_experiments), keyed by (sessionId, experimentId). The Replays filter and the AI Analyst join it to events, conversions, and replays on sessionId; the Experiments dashboard compares variants per device (deviceId). No variant field is needed on downstream events.

02Sticky by session, not by user

First assignment per (session, experimentId) wins. A later call with a different variant is a no-op.

Sumidata dedups on (sessionId, experimentId) - the first assignment the server sees for a session+experiment pair is the one that sticks. A second call with a different variant for the same experiment in the same session returns { status: "skip" } and writes nothing. This protects analysis from:

• Flicker-reassignment when a flag evaluator returns different values on re-mount.
• Double-fires when the SDK re-runs on route change.
• A client-side and a server-side assignment for the same session that arrive one after the other - two requests that arrive at the same moment can both be stored.

03From the browser

One push call. The SDK fills in deviceId and sessionId automatically.

Call identifyExperiment as soon as your flag framework has resolved assignments for the current user - ideally before the first product event fires. The join is not ordered in time: a sessionId join matches every event in that session, including ones sent before the call, and the Experiments dashboard credits a variant with all purchases its devices made in the selected period (and, in the events-by-variant breakdown, all their events). A session whose call never reaches Sumidata (the visitor left first) is not in the experiment.

// Called once per session, as soon as your A/B framework resolves assignments
Sumidata.push('identifyExperiment', [[
  { id: 'pricing-v3', variant: 'control' },
  { id: 'onboarding-flow', variant: 'short' },
  { id: 'hero-copy', variant: 'v2' },
]])

04From the server

Post the same shape from your backend when assignments are resolved server-side.

Useful when your flag framework runs in SSR, edge workers, or an internal feature-flag service, and you want to register variants without waiting for the client. You need the browser's sessionId and deviceId - get them from the SDK on the client first (see Linking browser session to server), store them on the user row, then call this endpoint.

curl "https://api.sumidata.io/sdk/session-experiments" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"projectId":"…","deviceId":"…","sessionId":"…","experiments":[{"id":"pricing-v3","variant":"control"}]}'

await axios.post('https://api.sumidata.io/sdk/session-experiments', {
  projectId: process.env.SUMIDATA_PROJECT_ID,
  deviceId: user.sumidataDeviceId,
  sessionId: user.sumidataSessionId,
  experiments: flags.resolveForUser(user.id),
})

{ "status": "ok" }     // new assignments inserted
{ "status": "skip" }   // every (sessionId, experimentId) was already registered
{ "status": "quota_exceeded" }   // monthly session quota used up - nothing stored

05Integration recipes

A one-call recipe for each of the common flag frameworks.

The pattern is always the same: resolve the variant with your flag service, then hand the result to Sumidata. Examples in the browser SDK; swap to the server endpoint if the decision happens on your backend.

GrowthBook

import { GrowthBook } from '@growthbook/growthbook'

const gb = new GrowthBook({ /* … */ })

gb.setTrackingCallback((experiment, result) => {
  Sumidata.push('identifyExperiment', [[
    { id: experiment.key, variant: result.key }
  ]])
})

Statsig

import { StatsigClient } from '@statsig/js-client'

const statsig = new StatsigClient({ /* … */ })
await statsig.initializeAsync()

const pricing = statsig.getExperiment('pricing-v3')
Sumidata.push('identifyExperiment', [[
  { id: 'pricing-v3', variant: pricing.get('variant', 'control') }
]])

LaunchDarkly

import * as LDClient from 'launchdarkly-js-client-sdk'

const ld = LDClient.initialize(clientId, { kind: 'user', key: user.id })
await ld.waitForInitialization()

const variant = ld.variation('pricing-v3', 'control')
Sumidata.push('identifyExperiment', [[
  { id: 'pricing-v3', variant }
]])

Optimizely

import { createInstance } from '@optimizely/optimizely-sdk'

const optimizely = createInstance({ sdkKey: '…' })
await optimizely.onReady()

const decision = optimizely.createUserContext(user.id).decide('pricing-v3')
Sumidata.push('identifyExperiment', [[
  { id: decision.flagKey, variant: decision.variationKey }
]])

06Reading the results back

Ask the AI Analyst in plain English, or open the Experiments dashboard and the Replays filter.

Sumidata intentionally does not expose a read endpoint for assignments - you almost never want the raw rows, you want the analysis built on top. Three ways to get that:

AI Analyst

Plain English. Works out of the box once any assignments are registered.

revenue and conversion rate by variant for pricing-v3, last 14 days

which onboarding-flow variant has the lowest drop-off on step 2?

Experiments dashboard

Under Analytics → Experiments you get a list of every experiment in your window ranked by sales contrast (the relative gap in purchase conversion between your best- and worst-converting variant; experiments that still need more data come last), a per-variant detail view (devices, sessions, purchases, revenue, conversion with a 95% confidence interval, average session length, and events per session), and an events-by-variant breakdown showing how each tracked event fires per device across variants. The unit is the device: each device counts toward the variant it was assigned, with all its purchases in the selected period (and all its events in the events-by-variant breakdown); sessions, average session length, and events per session count only the sessions that registered the experiment. A device assigned more than one variant of the same experiment is excluded. No setup - it populates as soon as assignments are registered.

Replay filtering

In the Replays dashboard, the Experiment and Variant selects scope the session list to a single experiment and variant (the options are the experiments with assignments in the last 90 days). Each variant row in the experiment detail deep-links here via ?experimentId=&variant=, which pre-fills both selects - great for qualitative review, watching five "treatment" sessions against five "control" back-to-back.

07Limitations

What Sumidata won't do for you - fill these in with your own tooling.

• No read endpoint for assignments. There is no GET /sdk/session-experiments. If you need the raw variant for a given session, ask the AI Analyst for the session_experiments rows.
• Sample sizes and confidence intervals, not verdicts. The Experiments dashboard shows per-variant conversion with 95% confidence intervals and the sample sizes behind every number. It does not issue significance verdicts - no "winner", no "significant". Whether a difference is real enough to act on stays with you and your decision tooling. Sumidata is the assignment logger, not the decision engine.
• No webhooks on new assignments. The registration is synchronous and returns a status, but downstream systems aren't notified.
• Session-scoped stickiness. If you need lifetime stickiness, resolve the variant in your flag service on every session and re-register - the first per-session assignment will match what you chose.
• 36-month retention. Assignments are deleted 36 months after they were recorded, the same as the rest of your analytics data, on every plan.

08AI-agent quick reference

Drop this into an agent prompt when automating experiment setup and analysis.

concept: Sumidata records which variant was shown; your flag service decides who sees what.

register:
  browser: Sumidata.push('identifyExperiment', [[{ id, variant }, …]])
  server: POST /sdk/session-experiments { projectId, deviceId, sessionId, experiments: [{ id, variant }] }
  response: { status: "ok" } (inserted) | { status: "skip" } (already registered) | { status: "quota_exceeded" } (nothing stored)

stickiness:
  dedup_key: (sessionId, experimentId) - first-wins
  changing_variant: requires a new session (logout / reset / 30 min without activity / 4 h cap) - in an open page the SDK re-registers the last variants on the new session
  scope: session-bound, not user-bound

timing:
  register_before: as early as possible - a session with no assignment row is not in the experiment
  retroactive: joins are not time-ordered - earlier events in the same session (dashboard: same device, selected period) are included

analysis:
  join: events JOIN session_experiments ON sessionId (+ projectId)
  no_variant_field_on_events: true - never stamp variant on event payloads; the join handles it
  dashboard: Analytics → Experiments - list ranked by sales contrast, per-variant detail (conversion + 95% CI), events-by-variant; unit = device, devices with more than one variant excluded
  session_filter: Replays accepts Experiment + Variant; deep link /dashboard?to=replays&experimentId=&variant=
  ai_analyst: "revenue by variant for pricing-v3 last 14 days"

limits:
  - no GET endpoint for raw rows - inspect via the AI Analyst
  - CI + sample sizes shown, but no significance verdict - decision stays with you
  - no webhooks on new assignments
  - assignments deleted 36 months after they were recorded