A multi-tenant ticketing platform: zero-downtime releases and a payment path tested against double charges

The situation

Many organizations sell tickets from one codebase and one production database. Each organization is a tenant: its events, orders, payouts and staff are scoped to it, and the access guard resolves the tenant from the record being touched, never from what the request claims. Buyers pay through Stripe and receive numbered QR tickets that door staff scan. In September 2026 production held a few hundred user accounts.

There is no maintenance window. A deploy migrates the database first and the serverless host switches traffic a few minutes later, so for those minutes the previous build runs against the new schema. A careless release can take a buyer's money and issue nothing, or sell one seat twice.

What we did

Only the pipeline can deploy production

The hosting platform's Git integration deployed main on every push, so CI raced the deploy instead of gating it. We turned off Git deploys of main in the platform config. One workflow job now deploys production, and it needs both the unit and the browser suites. It applies migrations before it triggers the build, so a failed migration stops the release while the previous build keeps serving. An illustration of the shape:

deploy:
  needs: [unit-tests, browser-tests]      # both suites, not only the fast one
  if: github.event_name == 'workflow_dispatch' ||
      (github.event_name == 'push' && github.ref == 'refs/heads/main')
  concurrency: { group: deploy-production, cancel-in-progress: false }
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - run: npm ci
    - name: Migrate production first
      run: npx prisma migrate deploy
      env: { DATABASE_URL: "${{ secrets.PROD_DIRECT_URL }}" }
    - name: Deploy
      run: npx vercel deploy --prod --yes --token="$VERCEL_TOKEN"
      env: { VERCEL_TOKEN: "${{ secrets.VERCEL_TOKEN }}" }

A job is skipped when a job it needs was skipped, so each suite lists workflow_dispatch in its own condition. npm audit at high is a CI gate too; it retries only on registry endpoint errors and fails at once on a real advisory.

Within days the gate held: a change reached main with a failing browser suite, the deploy job was skipped, and production stayed on the previous build.

Tests that prove which build they tested

One browser run spent its first minute testing the build from before the merge. A health endpoint now returns the deployed commit SHA, baked in at build time, but only to a caller holding a shared key compared in constant time. CI polls for up to ten minutes until the deployed SHA matches the commit under test, and on timeout the error names the likely cause. A green browser run refers to the build just deployed.

A migration history that can rebuild the database

The first CI run that applied every migration to an empty database showed the history did not describe the real schema: part of a subsystem had been applied with prisma db push. We generated one baseline migration from the confirmed live schema, proved a fresh Postgres rebuilt from it with zero drift, and marked it applied on each environment. Every CI run now migrates an empty database and diffs the result against the schema file with --exit-code, so an unmigrated edit fails the next run.

Migrations that the old code can live with

The September 2026 release carried eight migrations. Every one adds or relaxes and none removes, so rolling back means redeploying old code, not reversing a migration:

• New per-line money columns are nullable with no default and no backfill. NULL means "written before this release", and every reader falls back to the earlier calculation for those rows.
• Before the release, a database client generated from the previous production commit ran against the migrated schema.
• The previous commit was tagged, and the written rollback was one data step (one new enum value would make the old client throw) plus a redeploy of that tag, with the schema left alone.

Anything a migration backfills can go stale during the deploy window, because the old code is still writing. We replaced ticket numbers taken from a sold count (which reissues a number after a refund) with a backfilled counter per ticket tier, and the old code kept selling without touching it. Fulfillment now raises the counter to the highest issued number on every sale, under the row lock it already holds. An illustration:

-- runs inside the sale's transaction, after SELECT ... FOR UPDATE on the tier
UPDATE ticket_tier t
SET last_sequence = GREATEST(
  t.last_sequence,
  COALESCE((SELECT MAX(k.sequence_no) FROM ticket k WHERE k.tier_id = t.id), 0)
)
WHERE t.id = $1;

On a clone, the unfixed code threw a unique-constraint error on the first sale after the window; the fixed code issued the next number.

The release applied all eight migrations in under 20 seconds. The new build went live about three minutes later, so for about 2.5 minutes the old code served the new schema, with zero errors and production answering 200 throughout. The rollback was not needed.

A payment path tested against double charges

Stripe delivers webhooks at least once, sometimes concurrently. The handler records each event id in a ledger with a unique key before any slow side effect. Fulfillment locks the order, then the ticket tiers in sorted id order, and re-checks "already paid" under the lock. Errors that mean an order can never succeed (sold out after capture, amount mismatch) have their own type and trigger a refund; any other error makes Stripe retry. Refunds carry a Stripe idempotency key and a compare-and-set on the ticket row.

The design is that a retry cannot charge a card and issue nothing. We tested it with separate operating-system processes: one winner in each of five rounds of six, and a webhook event sent three times concurrently changed nothing.

Results

We also turned on certificate verification for the app's database connections; the method is in the article below.

Related reading

• Verified Postgres TLS when no host chains to a public root