Historical backfill

Replay past orders - and, optionally, past sessions with their UTMs - so reports start with full revenue context from day one. Events keep their original timestamps; first-time-buyer flags stay consistent across the replay.

01How backfill works

Two endpoints cover the whole import - one for sessions, one for events. Sessions reject a reused sessionId; orders are deduplicated by orderId.

Sumidata's ingest API is the same surface the browser SDK uses - there is no separate bulk-import pipeline. Backfill means replaying historical rows through two endpoints with caller-supplied UUIDs:

• POST /sdk/sessions/import - optional. Reconstructs a historical session with its UTM snapshot, so session counts in campaign reports cover the backfilled period.
• POST /sdk/ingest with source: 'import' - the same endpoint the SDK uses, with relaxed validation that makes sessionId and replayId optional.

Send each order once, with its products in items. If your export has one row per order line, group the rows by order before sending - a purchase with product fields at the top level is skipped, and a second event with the same orderId is skipped as a duplicate.

const byOrder = new Map<string, OrderLine[]>()
for (const line of lines) byOrder.set(line.orderId, [...(byOrder.get(line.orderId) ?? []), line])

const events = [...byOrder].map(([orderId, orderLines]) => ({
  name: 'purchase',
  orderId,
  timestamp: orders.get(orderId).paidAt.getTime(),
  totalAmount: Number(orders.get(orderId).total),  // order net - not the sum of line prices
  currency: orders.get(orderId).currency,
  items: orderLines.map((l) => ({
    productId: String(l.sku),
    productName: l.name,
    productCategory: l.category,
    quantity: Number(l.qty),
    unitPrice: Number(l.price),  // per-unit price
  })),
}))

Each item needs productId as a non-empty string; quantity must be a whole number from 1 to 4294967295 (1 when omitted) and unitPrice a number from 0 to below 10^16 (0 when omitted). Convert before sending: CSV values and Postgres numeric columns arrive as strings, and a numeric productId or a string quantity / unitPrice drops the item. An order holds at most 200 items. Dropped items are listed in the response's warnings; the order itself is stored without them. Revenue comes from totalAmount only - unitPrice never changes it.

02Timestamps

Always send timestamp - never let ingest-time stand in for event-time.

When you replay historical orders, the time the request reaches Sumidata is not the time the sale happened. If you omit timestamp, a year of orders collapses into today and every funnel, cohort, and retention chart breaks.

Send timestamp as Unix milliseconds matching when the business event actually occurred - orders.paid_at, invoice.settled_at, whatever your system of record stores. No min/max is enforced by the server; timestamps decades in the past or minutes in the future are both accepted. Dashboard reports look back at most 36 months by the order's timestamp, so older orders are stored but not counted in them. Stored orders and their products are deleted 36 months after they were ingested.

const event = {
  name: 'purchase',
  orderId: order.id,
  totalAmount: order.total,
  timestamp: order.paidAt.getTime(),  // original event time, in ms
}

03Importing historical sessions

Reconstruct historical sessions with their UTMs, so campaign reports show sessions for the backfilled period.

Orders do not read UTMs from the session. Each order stores a UTM snapshot at ingest: the utmSource / utmMedium / utmCampaign / utmTerm / utmContent you put on the event win; each one left empty is filled, field by field, from the latest campaign Sumidata holds at import time for the request's externalUserId (written by identify), or else for its deviceId - not the campaign from the order's date. For a backfill, put the historical UTMs directly on each purchase event. Orders still without UTMs after this fill are reported under (direct).

Importing sessions fills the session side of campaign reports (sessions and visitors per campaign, by startedAt). Each call creates the session row and, if any UTM parameter is present, a device_attribution entry for its deviceId. An order later sent with that deviceId picks up the UTMs of the device's latest session that carried UTMs (by start time) for any UTM field it leaves empty, unless the customer already has an identify campaign under that externalUserId.

import { randomUUID } from 'crypto'

const sessionId = randomUUID()

await axios.post('https://api.sumidata.io/sdk/sessions/import', {
  projectId: process.env.SUMIDATA_PROJECT_ID,
  sessionId,
  deviceId: order.deviceId ?? '00000000-0000-0000-0000-000000000000',
  externalUserId: order.userId,
  startedAt: order.sessionStartedAt.getTime(),
  utmSource: order.utmSource,
  utmMedium: order.utmMedium,
  utmCampaign: order.utmCampaign,
  referrer: order.referrer,
  landingPage: order.landingPage,
})

{ "sessionId": "…", "status": "imported" }

04Importing historical orders

Replay orders in chronological order, carrying the session ID if you imported one.

Expose an admin-only endpoint that iterates your order table and posts each row to /sdk/ingest with source: 'import'. Process orders chronologically - this is essential for correct isPrimarySale flags (see section 05).

@Get('/admin/sumidata/sync')
async sync(@Query() q: { pwd: string, startDate: string, endDate: string }) {
  if (q.pwd !== process.env.ADMIN_PASSWORD) throw new UnauthorizedException()

  const from = new Date(q.startDate + 'T00:00:00Z').getTime()
  const to = new Date(q.endDate + 'T23:59:59Z').getTime()

  return this.syncService.run({ from, to })
}

05Keeping isPrimarySale honest

Track which users got a 'first purchase' flag during this run - don't compute it against the live DB row-by-row.

If you evaluate "is this the user's first purchase?" against the live DB for every replayed order, every order in the range looks like a repeat (because all orders are already in the DB). Fix this with two sets:

• Prior purchasers - users who had a purchase before the replay range starts. Query once at the beginning.
• Purchasers in this run - users who've had their first order flagged during the current replay. Track in-memory as you iterate chronologically.

A user is a primary sale if they're in neither set when their earliest in-range order is processed.

async run({ from, to }: { from: number, to: number }) {
  const priorPurchasers = await orders.distinctUserIds({ paidBefore: from })
  const purchasersInSync = new Set<string>()
  const errors: any[] = []
  let sent = 0

  const rows = await orders.find({ paidAt: { between: [from, to] }, orderBy: 'paidAt ASC' })

  for (const order of rows) {
    const isPrimarySale =
      !priorPurchasers.has(order.userId) && !purchasersInSync.has(order.userId)

    if (isPrimarySale) purchasersInSync.add(order.userId)

    try {
      await ingest({
        projectId: process.env.SUMIDATA_PROJECT_ID,
        deviceId: '00000000-0000-0000-0000-000000000000',
        externalUserId: order.userId,
        source: 'import',
        events: [{
          name: 'purchase',
          orderId: order.id,
          totalAmount: order.total,
          timestamp: order.paidAt.getTime(),
          isPrimarySale,
        }]
      })
      sent++
    } catch (e) {
      errors.push({ orderId: order.id, error: String(e) })
    }
  }

  return { totalProcessed: rows.length, sent, errors: errors.slice(0, 20) }
}

06Pre-flight checklist

Five checks before you pull the trigger on a large replay.

const userIds = [...new Set(rows.map((r) => r.userId))]
const productIds = [...new Set(rows.map((r) => r.productId))]

const [usersMap, productsMap] = await Promise.all([
  users.findMapByIds(userIds),
  products.findMapByIds(productIds),
])

SELECT toDate(timestamp) AS day,
       count()                 AS conversions,
       sum(totalAmount)         AS revenue
FROM events
WHERE source = 'import' AND orderId != ''
  AND timestamp BETWEEN '2025-01-01' AND '2025-12-31'
GROUP BY day ORDER BY day

07Updating a backfilled order

Re-sending a conversion with a changed field is skipped by dedup. Here is what to do instead.

The dedup key orderId lasts as long as the order is stored - 36 months after it was ingested. You cannot overwrite a stored conversion by re-sending the same orderId with different fields - the retry is skipped as Duplicate conversion: … and the stored row is left untouched.

Two options, depending on why you need the update:

• You added a new field to your schema (e.g. you started tracking couponCode): for future orders, just start sending it. For history, the old events stay without the field - this is normal and expected, and reports handle missing values gracefully.
• A field was wrong (e.g. totals miscalculated in a historical export): contact support to purge the affected orderIds from events, purchase_items and dedup_keys, then re-run backfill for that range. Stored orders cannot be deleted self-serve today.

08AI-agent quick reference

Hand this checklist to an agent running an autonomous import.

goal: Replay historical orders into Sumidata, preserving timestamps and first-purchase flags.

endpoints:
  sessions:
    path: POST /sdk/sessions/import
    use_when: you have historical UTMs to preserve
    idempotency: caller-supplied sessionId; 400 if it exists
  events:
    path: POST /sdk/ingest
    body: { projectId, deviceId, externalUserId, source: "import", events: [...] }
    idempotency: orderId per project; a repeat is skipped, returned in response.skipped

per_event_required: [name, timestamp (ms)]
per_conversion_required: [orderId, totalAmount]
per_conversion_recommended: [currency, isPrimarySale, items (one event per order, products inside, max 200)]

ordering: process orders chronologically - required for correct isPrimarySale
primary_sale_rule: |
  isPrimarySale = user NOT in priorPurchasers AND NOT in purchasersInCurrentRun
  priorPurchasers   = distinct userIds with a paid order before range.start
  purchasersInCurrentRun = in-memory set, add userId when flagging primary

pacing:
  batch_size: 1–50 events per envelope (a duplicate is skipped alone in response.skipped; the rest still stores)
  throughput: few hundred requests/sec per project; pause 50–100ms every ~500 requests
  retry: 4xx = log and skip; 200 with status "quota_exceeded" = nothing stored, resend later; 5xx / network = exponential backoff; dead-letter after N retries

verify:
  - GET /api/partner/events?q[source__eq]=import - totalCount matches source DB
  - SQL: COUNT(*) and SUM(totalAmount) by day WHERE source='import' matches source DB
  - spot-check isPrimarySale by user - only one TRUE per user across the replay