title: Prod Deployment Runbook status: active last-updated: 2026-06-11 authors: [@amalkrsihna] audience: [operator, backend, frontend] abstract: > Step-by-step guide to promote code + manifest changes from release/staging to release/prod via the operator-specified four-branch flow. Per-service ordering, image-promotion details, migration handling, validation, and rollback. companion: [prod-launch-runbook, prod-snapshot-2026-06-11]

Prod Deployment Runbook¶

Companion: prod-launch-runbook, prod-snapshot-2026-06-11, incident-debugging.

Pre-launch tasks (PG / Redis / env vars) MUST be done first. See prod-launch-runbook.

0. Context¶

Production runs ~20 days behind staging today. This runbook ships today's release/staging HEAD onto prod via a four-branch flow that mirrors the operator's specified branching strategy:

                          ┌─── PR 1 ───┐
release/staging  ──► staging-to-prod   ──►  prod-integration-branch  ──► release/prod
                                            ▲
                              cut --no-track │
                              off origin/    │
                              release/prod   │
                                             │
                                       PR 2 ─┘

PR 1: staging-to-prod-release → prod-integration-branch
PR 2: prod-integration-branch → release/prod

Both PRs admin-merged in order. CI fires only on the final release/prod push.

1. Pre-flight (mandatory)¶

Run these before cutting any branch. Skip none of them.

Check	How	Expected
Prod launch runbook §3.1–3.6 complete	re-read `prod-launch-runbook` §3	✅ each step
GitHub `production` env has required reviewers	`gh api repos/JetaFutures/ftl-{backend,frontend}/environments/production --jq '.protection_rules'`	at least one reviewer
`release/staging` HEAD is green	`gh run list --branch release/staging --limit 3`	last 3 = `success`
Staging is healthy	`curl https://ftl-stg-api.kindfield-0bb54ed4.centralindia.azurecontainerapps.io/health`	200
All staging migrations applied	`psql $STAGING_DATABASE_URL -c 'SELECT version,dirty FROM schema_migrations ORDER BY version DESC LIMIT 1;'`	latest, `dirty=false`
No active staging incident	check ops channel	clear
No WC26 fixture in next 30 min	Sportmonks schedule	none

Shared ACR

Prod and staging share ftlstgacrcin in ftl-stg-rg-cin. The same image tag (Git SHA) is pushed once on the staging build and pulled by both envs. No "promote image" step is needed — release/prod CI re-tags / re-uses the same SHA, not a fresh build.

2. Branching strategy¶

Four branches involved per deploy cycle. Names are examples; use the dated naming convention.

release/staging                       (source — already at the desired SHA)
release/prod                          (target — currently at fea5dc41…)
prod-integration-<YYYY-MM-DD>         (NEW, cut --no-track off origin/release/prod)
staging-to-prod-release-<YYYY-MM-DD>  (NEW, cut --no-track off origin/release/staging)

The two new branches are merged in order so the change moves staging→prod through a reviewable seam.

3. Cut both new branches¶

cd /Users/amal/Projects/02-fifa26-trading-game/claude-code/repos/ftl-backend
git fetch origin --prune

# 1. Prod integration — cut from CURRENT prod
git checkout --no-track -b prod-integration-2026-06-11 origin/release/prod

# 2. Staging-to-prod release — cut from CURRENT staging
git checkout --no-track -b staging-to-prod-release-2026-06-11 origin/release/staging

# Push both
git push -u origin prod-integration-2026-06-11
git push -u origin staging-to-prod-release-2026-06-11

Repeat the same cuts in ftl-frontend if frontend changes ship together. (Today's session shipped FE goal-emoji marker; that's already on FE release/staging.)

4. Open + merge PR 1 (staging-to-prod → prod-integration)¶

This is the "diff review" gate — every commit between current prod and current staging shows up in a single PR.

CLIPortal (GitHub UI)

PAT=$(git remote get-url origin | grep -oE "ghp_[A-Za-z0-9_]+")
GH_TOKEN="$PAT" gh pr create \
  --repo JetaFutures/ftl-backend \
  --base prod-integration-2026-06-11 \
  --head staging-to-prod-release-2026-06-11 \
  --title "Promote release/staging → prod-integration 2026-06-11" \
  --body "$(cat <<'EOF'
## Summary

Promotes the entire `release/staging` HEAD to `prod-integration-2026-06-11`
for review before final cutover to `release/prod`.

## Changes since current prod

See the file diff. Notable surface areas:
- Sportmonks ticker (timeline include, micro_bump decay change)
- Position close cooldown (per-position from opened_at_ms)
- Telemetry sampler (ErrorPreservingProcessor at 1%)

## Test plan

- [x] All staging tests green (already passed for the release/staging head)
- [ ] Reviewer sign-off
- [ ] Then PR 2 (prod-integration → release/prod) opens.

Author: @amalkrsihna
EOF
)"

https://github.com/JetaFutures/ftl-backend/compare/prod-integration-2026-06-11...staging-to-prod-release-2026-06-11
Click Create pull request.
Title: "Promote release/staging → prod-integration 2026-06-11".
Body: paste the same template above.
Click Create pull request.

Review carefully. Compare to prod-snapshot-2026-06-11.md §5 to confirm every expected change is in the diff.

Merge:

GH_TOKEN="$PAT" gh pr merge <PR-number> \
  --repo JetaFutures/ftl-backend \
  --merge --admin --delete-branch

Risk: PR 1 doesn't trigger CI (no protected base) → first chance to catch a build break is at PR 2. Mitigation: before opening PR 2, manually run go build ./... and go test ./... against the merge head on a clean clone. Or rely on the fact that release/staging CI was already green at the source SHA.

5. Open + merge PR 2 (prod-integration → release/prod)¶

This is the final gate. The production GitHub Environment kicks in and pauses for the required reviewer.

CLIPortal (GitHub UI)

GH_TOKEN="$PAT" gh pr create \
  --repo JetaFutures/ftl-backend \
  --base release/prod \
  --head prod-integration-2026-06-11 \
  --title "release/prod: 2026-06-11 cutover" \
  --body "$(cat <<'EOF'
## Summary

Promotes prod-integration-2026-06-11 to release/prod, triggering the
Deploy backend to prod CI workflow.

Includes the change list reviewed in PR 1 (#<PR-1-number>).

## Test plan

- [ ] CI Deploy backend to prod completes (paused on `production` env approval)
- [ ] Operator approves the production environment gate
- [ ] Migrations job completes (Section 7 below)
- [ ] All three Container Apps roll
- [ ] Health smoke passes
- [ ] Post-deploy verification (Section 8 below)
EOF
)"

https://github.com/JetaFutures/ftl-backend/compare/release/prod...prod-integration-2026-06-11
Create pull request → title + body as above.
Wait for review approval.

Merge:

GH_TOKEN="$PAT" gh pr merge <PR-2-number> \
  --repo JetaFutures/ftl-backend \
  --merge --admin --delete-branch

This triggers deploy-prod.yml. The workflow will pause on the production environment approval gate. Approver receives a GitHub email + notification.

Portal (GitHub UI) — approve the gate

Open the workflow run https://github.com/JetaFutures/ftl-backend/actions
The "Deploy backend to prod" run is at status Waiting with a "Review deployments" button.
Click Review deployments → tick production → optionally comment → click Approve and deploy.

The pipeline continues. Watch:

Job	Expected duration	Notes
Lint + test	2–3 min	reuses Go modules cache
Build + push images (4 services)	4–6 min total	api, ws, flusher, migrate
Run database migrations	1–2 min	on Container Apps Job `ftl-prd-migrate`
Roll Container App (api)	1–2 min	rolling revision
Roll Container App (ws)	1–2 min	clients reconnect
Roll Container App (flusher)	1–2 min	singleton
Health smoke	30 s	`curl /health` on api + ws
Mark deployment complete	<30 s	GitHub deployments API status

Total: ~10–20 min.

6. What about Frontend?¶

If frontend also has changes (today's session shipped the goal-emoji marker), repeat Steps 3–5 for ftl-frontend. Static Web App build runs entirely in CI; on success it uploads to the prod SWA. No replicas to roll.

Frontend has its own deploy-prod.yml reading secrets.VITE_APPLICATIONINSIGHTS_CONNECTION_STRING_PROD — see github-env-audit-2026-06-11.md MISSING SECRET ALERT. Either rename the workflow reference to the existing repo-level secret, or set the _PROD variant on the production environment. Otherwise: build succeeds with an empty AppInsights conn string baked in, frontend ships with broken telemetry.

7. Database migration handling¶

The Run database migrations job runs ftl-prd-migrate (Container Apps Job, manual trigger). It uses golang-migrate against the prod PG via the DATABASE_URL env var.

Pre-deploy: prod schema is at the version corresponding to the May-22 image. Migrations applied between then and release/staging HEAD will run on this step.

Risk:

Migration that touches a large table can block reads/writes for minutes.
Migration dirty=true after partial failure blocks future migrations until manually fixed.

Mitigation:

# BEFORE the deploy: list migrations that will apply
diff <(ls ftl-backend/migrations/ | sort) \
     <(psql $PROD_DATABASE_URL -c 'SELECT version FROM schema_migrations ORDER BY version' -t)

If anything looks scary (DROP COLUMN, ALTER COLUMN TYPE on a user-data column), STOP. CLAUDE.md "Migration safety" rule #1 prohibits these without an ADR. Run a manual EXPLAIN against staging first.

After deploy:

psql $PROD_DATABASE_URL -c 'SELECT version, dirty FROM schema_migrations ORDER BY version DESC LIMIT 5;'

Expect: latest version matches the highest file under ftl-backend/migrations/, dirty=false on every row.

If dirty=true:

# Identify what failed (check the migrate job log)
GH_TOKEN="$PAT" gh run view <prod-deploy-run-id> --repo JetaFutures/ftl-backend --log

# Manually flip dirty=false ONLY AFTER confirming the failed migration is non-destructive
# and the schema is in a consistent state
psql $PROD_DATABASE_URL -c "UPDATE schema_migrations SET dirty=false WHERE version=<failed-version>;"

8. Post-deploy validation¶

Run all of these. Fail-fast on any.

8.1 Revision + image SHA¶

for app in ftl-prd-api ftl-prd-ws ftl-prd-flusher; do
  AZURE_CONFIG_DIR=~/.azure-ftl az containerapp show -g ftl-prd-rg-cin -n "$app" \
    --query '{rev:properties.latestRevisionName, image:properties.template.containers[0].image, traffic:properties.configuration.ingress.traffic}' -o json
done

Expected: each image ends with the SHA matching git rev-parse origin/release/prod. Each rev is a new revision number. Traffic 100 % on latest.

8.2 Health endpoints¶

curl -sw "\nHTTP %{http_code} | %{time_total}s\n" \
  https://prd-api.ftljeta.cloud/health    # or the prod ingress FQDN

curl -sw "\nHTTP %{http_code}\n" \
  https://prd-ws.ftljeta.cloud/health

Both expected 200, response < 500 ms.

8.3 Replica state¶

for app in ftl-prd-api ftl-prd-ws ftl-prd-flusher; do
  AZURE_CONFIG_DIR=~/.azure-ftl az containerapp replica list \
    -g ftl-prd-rg-cin -n "$app" \
    --query '[].{state:properties.runningState, restarts:properties.containers[0].restartCount}' -o json
done

Expected: every replica Running, restartCount = 0.

8.4 Error logs¶

AZURE_CONFIG_DIR=~/.azure-ftl az monitor log-analytics query \
  -w 70e5df86-b238-4a05-a4a3-503ffae12533 \
  --analytics-query "ContainerAppConsoleLogs_CL | where TimeGenerated > ago(10m) | where Log_s contains 'panic' or Log_s contains 'fatal' or Log_s contains 'level=error' | take 30" -o tsv

Expected: empty.

8.5 Env vars present (sanity)¶

for app in ftl-prd-api ftl-prd-ws ftl-prd-flusher; do
  AZURE_CONFIG_DIR=~/.azure-ftl az containerapp show -g ftl-prd-rg-cin -n "$app" \
    --query "properties.template.containers[0].env[?name=='SPORTMONKS_LEAGUE_IDS' || name=='OTEL_TRACE_SAMPLE_RATIO' || name=='APPLICATIONINSIGHTS_CONNECTION_STRING']" -o json
done

Expected: all three vars present on each app with values from prod-launch-runbook.md §3.3.

8.6 Smoke: complete user journey¶

Open the prod frontend ftl.jetafutures.com. As a test user:

Sign in with Google.
Open a player chart (Trade page).
Open one position (long, 0.1 lot).
Verify price chart updates within 10 s.
Wait 180 s, then close the position.
Verify P&L applied to wallet.

If any step fails → rollback (Section 9).

8.7 Cost check after 1 h¶

AZURE_CONFIG_DIR=~/.azure-ftl az consumption usage list \
  --start-date $(date -u +%Y-%m-%d) \
  --end-date $(date -u +%Y-%m-%d) \
  --query "[?resourceGroup=='ftl-prd-rg-cin']" -o json | \
  jq 'group_by(.meterCategory) | map({service: .[0].meterCategory, total_INR: ([.[].pretaxCost | tonumber] | add)})'

Expected: small increase over baseline. If LAW shows > 1 GB after 1 hour, something is over-logging. Sampler may not be in effect (OTLP unset, so this would only be container stdout).

9. Rollback¶

If smoke fails or replicas crash-loop:

9.1 Pin previous revision (fastest)¶

# Get previous revision
for app in ftl-prd-api ftl-prd-ws ftl-prd-flusher; do
  echo "=== $app ==="
  AZURE_CONFIG_DIR=~/.azure-ftl az containerapp revision list \
    -g ftl-prd-rg-cin -n "$app" \
    --query 'reverse(sort_by([?properties.active], &properties.createdTime))[1].name' -o tsv
done

# Activate the previous revision
for app in ftl-prd-api ftl-prd-ws ftl-prd-flusher; do
  AZURE_CONFIG_DIR=~/.azure-ftl az containerapp revision activate \
    -g ftl-prd-rg-cin -n "$app" --revision <prev-rev-name>
done

This restores the previous image + env in 30 s.

9.2 Forward fix via new PR (preferred)¶

For non-critical issues, cut a fix branch off prod-integration-2026-06-11 and re-run the four-branch flow with the fix included.

9.3 Database rollback¶

DO NOT do migrate down on prod. Per CLAUDE.md "Migration safety" rule, rollback via PITR or a new forward migration. If a destructive migration was applied by mistake:

# Get the timestamp from BEFORE the deploy
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server show \
  -g ftl-prd-rg-cin -n ftl-prd-pg-cin --query 'backup.earliestRestoreDate' -o tsv

# PITR restore
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server restore \
  -g ftl-prd-rg-cin -n ftl-prd-pg-cin-recovery \
  --source-server ftl-prd-pg-cin \
  --restore-time '<timestamp BEFORE deploy>' -o json

The restored server is ftl-prd-pg-cin-recovery — a new server. Re-point app DATABASE_URL to it, validate, then swap names if confident. This is a multi-hour operation; only invoke if the deploy genuinely broke data.

10. Cleanup after a successful deploy¶

Both new branches (staging-to-prod-release-... and prod-integration-...) auto-deleted on PR merge if --delete-branch flag was set.

If kept around, delete manually after 24 h to keep the branch list tidy:

git push origin --delete prod-integration-2026-06-11 staging-to-prod-release-2026-06-11

11. Risks summary¶

Risk	Mitigation
Wrong base on PR 1 (cut off main instead of release/prod)	Verify `git log origin/release/prod..prod-integration-2026-06-11` is empty before opening PR.
PR 2 merges without reviewer approval	`production` environment reviewer gate (Step 6 in prod-launch-runbook).
Migrate job leaves `dirty=true`	§7 manual recovery; do not deploy further until cleaned.
Image SHA mismatch (prod pulls wrong tag)	Hash compare in §8.1.
WS reconnect storm on roll	Inherent — clients reconnect with backoff. Acceptable for the deploy window.
Sportmonks API quota tripped by retry storm	Confirm `rate_limit.remaining > 1000` on first poll. CLAUDE.md security checklist #8.
Frontend ships with broken AppInsights	Fix `VITE_APPLICATIONINSIGHTS_CONNECTION_STRING_PROD` secret BEFORE deploy. See github-env-audit.