Offline upgrades for a festival bar's inventory system

The situation

The client runs bars at outdoor festivals: a few days at a site, several bar points, crates of stock arriving on supplier invoices, and complimentary drinks to account for. The stock and cash-flow system runs on a single small computer at the site. There is no reliable internet at the venue, so there are no registry pulls, no cloud database and no remote hands.

The system's ledger is append-only, enforced in the database by triggers that reject updates, deletes and truncates. Corrections are new rows pointing at the row they reverse. That keeps stock and money auditable, and it also means a bad migration cannot be patched by hand on site. Whatever the upgrade does to real data has to be right before the box leaves the office.

Alongside it we built a touch till for the bars, which drives a receipt printer and a card terminal. It is designed so a lost reply from the printer never causes a second receipt. This page is about the inventory system's upgrade path.

What we did

One tarball for a site with no internet

Each release is a single tarball: the application images, the vendored database, cache and proxy images saved with docker save, the compose file, an idempotent installer, the invariant library described below, and a VERSION file with the commit and build date. Nothing is pulled at the site. The installer generates secrets on first run only, detects the LAN address, issues a self-signed certificate for it, and is safe to run again.

A backup before every migration

Before migrating, the installer dumps the database and keeps the newest 30 dumps. An illustration of the pattern:

set -euo pipefail
backup="$BACKUP_DIR/inventory-$(date +%Y%m%d-%H%M%S).sql.gz"
if docker compose exec -T db pg_dump -U app -d app --clean --if-exists | gzip > "$backup"; then
  ls -1t "$BACKUP_DIR"/inventory-*.sql.gz | tail -n +31 | xargs -r rm -f
else
  rm -f "$backup"      # never keep a partial dump
  echo "warning: pre-migration backup failed" >&2
fi

The first line carries the weight. Without pipefail, the pipeline's status is gzip's, so a failed dump leaves a valid-looking, empty file that the retention step then keeps as the newest backup.

Upgrades rehearsed on the newest real backup

After the first festival we went through what had happened on site, and one finding was that our tests had only ever run on data we invented. A migration that is wrong for real data cannot be fixed at the venue.

So we built a local harness. It takes the newest production dump, restores it into a scratch database, snapshots row counts, runs alembic upgrade head, runs an invariant battery, compares row counts with a committed baseline of expected changes, and boots the API against the result. The festival's own dump was carried this way through fourteen migrations.

The battery checks, among other things, that no foreign key points at a missing row. It does not list the foreign keys by hand. It reads them all from the Postgres catalog (pg_constraint), so a table added next month is covered on the day it lands. The expected-change baseline is a plain text file next to the migrations, with one rule in its header: update it in the same commit as any migration that changes expected behavior. A row count that moves without a matching line in that file fails the rehearsal.

The same battery library ships inside the bundle and runs on the box after every install, against the data that is really there.

The rebuild ships the same way

The inventory system was later rebuilt on a fresh migration chain. It sets one transaction per migration from its first revision (the second fault below explains why) and refuses to start if the database URL is unset, so a misconfigured install never targets a placeholder. The rebuilt version now ships as an offline bundle too.

Results