title: Prod Launch Runbook status: active last-updated: 2026-06-11 authors: [@amalkrsihna] audience: [operator, ops] abstract: > Step-by-step ordered execution guide to take prod from the measured 2026-06-11 state to the WC26-ready state. Every step has CLI commands, Azure Portal UI walkthrough, pre-actions, post-actions, risk, mitigation, and rollback. Backup-first ordering. companion: [prod-snapshot-2026-06-11, cost-estimate-audit-2026-06-11]

Prod Launch Runbook¶

Companion: prod-snapshot-2026-06-11, cost-estimate-audit-2026-06-11, prod-deployment-runbook, incident-debugging, github-env-audit-2026-06-11.

READ THIS WHOLE PAGE BEFORE STARTING. Steps are ordered. Skipping one (especially backup) breaks rollback.

0. Context¶

prod-snapshot-2026-06-11.md measured eight outstanding items between current prod and WC26-ready state. This runbook executes them in safe order: backups first, then config-only changes, then SKU upgrades, then image roll, then verification. Each step is independent so partial completion is recoverable.

No code changes are required in this runbook. All code is already on release/staging; the image roll happens via the prod-deployment-runbook which is a separate sequence.

1. Pre-flight (do these ONCE before starting)¶

Item	Verify command	Expected
Azure CLI scoped to FTL account	`AZURE_CONFIG_DIR=~/.azure-ftl az account show --query '{user:user.name,sub:name}' -o tsv`	`jetafifa2026@gmail.com FootbalTradeLeage`
Two operator hands on the page	call ops partner	one runs, one watches
Active incident?	check PagerDuty / WhatsApp ops channel	clear
GitHub Actions queue empty	`gh run list --repo JetaFutures/ftl-backend --limit 3`	none `in_progress` against `release/prod`
WC26 fixture in next 90 min?	check Sportmonks schedule	if yes, postpone risky steps
Budget alert wired	Cost Management → Budgets → ftl-prd-rg-cin	active

Required reviewers gate is OFF on prod env

Both JetaFutures/ftl-backend and JetaFutures/ftl-frontend have a production GitHub Environment with no required reviewers. Any push to release/prod will auto-deploy without human gate. See github-env-audit-2026-06-11.md. Enable reviewers before Step 6.

2. Step ordering — first things first¶

1. BACKUP — Postgres + Redis snapshot           (lowest blast radius, enables rollback)
2. AUTOGROW — Enable PG autoGrow                (instant, no downtime)
3. ENV VARS — Set Container App env vars        (one-time, no replica roll yet)
4. RESIZE  — Postgres SKU B2ms → D2ds_v5        (online, ~5–10 min)
5. RESIZE  — Redis Basic C0 → Standard C1       (~30 s failover)
6. GH RBAC — Enable required reviewers on prod  (BEFORE first prod image roll)
7. ROLL    — Promote release/staging to prod   (separate runbook)
8. VERIFY  — Full post-launch checks            (Section 10)

Each numbered section below mirrors this order. Numbers match — Step 1 in §3.1, Step 2 in §3.2, etc.

3. The steps¶

3.1 Step 1 — Backup Postgres + Redis BEFORE anything¶

PG Flex has automatic PITR (7-day retention). Take an explicit on-demand backup so we have a named restore point at a known good time.

Pre-action: Note the current latest_restorable_time so you have a "before" mark.

CLIPortal

# Note current PITR baseline
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server show \
  -g ftl-prd-rg-cin -n ftl-prd-pg-cin \
  --query 'backup' -o json

# Trigger an on-demand backup (Flexible Server feature)
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server backup create \
  -g ftl-prd-rg-cin -n ftl-prd-pg-cin \
  --backup-name "pre-wc26-launch-$(date -u +%Y%m%dT%H%M)" \
  -o json

# Confirm the backup is queued / completed
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server backup list \
  -g ftl-prd-rg-cin -n ftl-prd-pg-cin -o table

Portal → search ftl-prd-pg-cin → click resource.
Left blade → Backup and restore.
Top bar → Backup now.
Backup name: pre-wc26-launch-YYYYMMDDTHHMM (UTC).
Click Backup.
Wait until the new row appears in the backup list with status Completed.

Redis Basic does not support snapshots; the only durable state in Redis is throwaway cache (instrument hashes are rehydrated from PG; dirty sets are forward-only). Skip Redis snapshot — there's nothing to preserve at this tier.

Post-action: Tag the backup name in the ops channel. Expected duration: 1–3 min on the current 64 GiB volume.

Risk: Backup competes for IOPS with live PG queries. Mitigation: PG B2ms has 240 baseline IOPS on the P6 volume; backup adds ~50 IOPS for a few minutes. No real users yet → no live workload to interfere with. Schedule for before the first WC26 kick-off, when no live activity.

Rollback if Step 1 fails: none needed — nothing changed yet. Read the az error, fix, retry.

3.2 Step 2 — Enable PG autoGrow¶

Current: storage.autoGrow = Disabled, fixed 64 GiB ceiling. The WC26 tournament forecast is ~9–12 GiB growth on a healthy run, so 64 GiB has 5× headroom — but a single runaway poller / bot regression can fill it in hours. Enable autoGrow so the disk extends on its own instead of HALTING writes.

Pre-action: Confirm Step 1 backup completed.

CLIPortal

AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server update \
  -g ftl-prd-rg-cin -n ftl-prd-pg-cin \
  --storage-auto-grow Enabled \
  -o json

Portal → ftl-prd-pg-cin → Settings blade → Compute and storage.
Find Storage auto-growth toggle. Currently Disabled.
Switch to Enabled.
Click Save at the bottom.
Wait for green notification "Updated successfully".

Risk: None at the toggle level (no downtime). Risk is structural — a runaway will keep growing until it hits the 16 TiB max for the tier, which would be a far worse bill before alerting fires. Mitigation: Combine autoGrow with Cost Management budget alert at ₹50K (already recommended in §13.1 of the cost report). Disk autoGrow + cost alert = both write-availability and bill control.

Rollback: Same command with --storage-auto-grow Disabled. Instant. No data effect.

Post-action: Re-verify with:

AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server show \
  -g ftl-prd-rg-cin -n ftl-prd-pg-cin \
  --query 'storage' -o json

Expected: "autoGrow": "Enabled".

3.3 Step 3 — Set Container Apps env vars¶

Three env vars need to land on all three apps so the next image roll picks them up. Setting these does not restart replicas; they take effect on the next revision.

Env var	Value	Why
`SPORTMONKS_LEAGUE_IDS` (plural)	`732`	WC26 only. Drops N_I from ~5,664 → ~830. Smaller Redis budget, lower bot CPU.
`OTEL_TRACE_SAMPLE_RATIO`	`0.01`	Insurance — activates the SDK sampler if/when OTLP endpoint is wired (post-launch decision).
`APPLICATIONINSIGHTS_CONNECTION_STRING`	secretref `appinsights-conn`	Mirror staging pattern. Captured by config.Load; ready for future direct exporter.

Also remove the legacy stale env var:

Env var	Action
`SPORTMONKS_LEAGUE_ID` (singular, value `271`)	Remove. Dead config key in current binary.

Pre-action: Confirm the Key Vault secret ftl-prd-kv-cin/appinsights-connection-string exists and Container Apps environment has it referenced as a Container App secret named appinsights-conn. Without this the env var write will fail.

# Verify KV secret exists
AZURE_CONFIG_DIR=~/.azure-ftl az keyvault secret show \
  --vault-name ftl-prd-kv-cin --name appinsights-connection-string \
  --query 'name' -o tsv
# expect: appinsights-connection-string

# Verify Container App secret exists on each app
for app in ftl-prd-api ftl-prd-ws ftl-prd-flusher; do
  AZURE_CONFIG_DIR=~/.azure-ftl az containerapp secret list \
    -g ftl-prd-rg-cin -n "$app" \
    --query "[?name=='appinsights-conn'].name" -o tsv
done
# expect: 'appinsights-conn' on each line

If appinsights-conn is NOT present on a Container App, add it from KV first:

CLIPortal

for app in ftl-prd-api ftl-prd-ws ftl-prd-flusher; do
  AZURE_CONFIG_DIR=~/.azure-ftl az containerapp secret set \
    -g ftl-prd-rg-cin -n "$app" \
    --secrets "appinsights-conn=keyvaultref:https://ftl-prd-kv-cin.vault.azure.net/secrets/appinsights-connection-string,identityref:system"
done

Portal → ftl-prd-api → Settings → Secrets.
+ Add → Key: appinsights-conn. Type: Reference a secret from Key Vault. Vault: ftl-prd-kv-cin. Secret: appinsights-connection-string. Click Add.
Repeat for ftl-prd-ws and ftl-prd-flusher.

Now apply the env vars on each app:

CLIPortal

for app in ftl-prd-api ftl-prd-ws ftl-prd-flusher; do
  AZURE_CONFIG_DIR=~/.azure-ftl az containerapp update \
    -g ftl-prd-rg-cin -n "$app" \
    --set-env-vars \
      "SPORTMONKS_LEAGUE_IDS=732" \
      "OTEL_TRACE_SAMPLE_RATIO=0.01" \
      "APPLICATIONINSIGHTS_CONNECTION_STRING=secretref:appinsights-conn" \
    --remove-env-vars SPORTMONKS_LEAGUE_ID
done

For each app (ftl-prd-api, ftl-prd-ws, ftl-prd-flusher):

Portal → ftl-prd-{api,ws,flusher} → Application → Containers.
Click the container row → Edit and deploy.
In the env-var grid:
Add row SPORTMONKS_LEAGUE_IDS = 732 (plain value).
Add row OTEL_TRACE_SAMPLE_RATIO = 0.01.
Add row APPLICATIONINSIGHTS_CONNECTION_STRING → Source: Reference a secret → choose appinsights-conn.
Remove the row SPORTMONKS_LEAGUE_ID (singular legacy).
Click Save → Create. A new revision is created.

Risk: --set-env-vars triggers a new revision automatically — this rolls the replicas. ~30 s of dropped WS connections during the roll. Clients reconnect automatically; the only side effect is everyone hitting user:reconnect simultaneously, which fires margin replay for every connected user. Mitigation: Stagger the apps: do flusher first (no ingress, no users notice), then api, then ws last (the one that drops WS). Wait 60 s between each.

Rollback: Re-run az containerapp update with the previous values, or pin the previous revision:

AZURE_CONFIG_DIR=~/.azure-ftl az containerapp revision activate \
  -g ftl-prd-rg-cin -n ftl-prd-api --revision ftl-prd-api--<previous-rev>

Post-action:

for app in ftl-prd-api ftl-prd-ws ftl-prd-flusher; do
  echo "=== $app ==="
  AZURE_CONFIG_DIR=~/.azure-ftl az containerapp show \
    -g ftl-prd-rg-cin -n "$app" \
    --query "properties.template.containers[0].env[?name=='SPORTMONKS_LEAGUE_IDS' || name=='OTEL_TRACE_SAMPLE_RATIO' || name=='APPLICATIONINSIGHTS_CONNECTION_STRING']" -o json
done

Expected: each app shows all three vars set.

3.4 Step 4 — Upgrade Postgres SKU `B2ms` → `D2ds_v5`¶

Burstable B2ms uses CPU credits (~40 % sustained baseline = ~0.8 vCPU eff). D2ds_v5 gives 2 dedicated vCPU continuously, removes the credit-cliff risk. Online resize, ~5–10 min, no app downtime.

Pre-action: 1. Step 1 backup completed. 2. App connections drain naturally — PG Flex keeps active connections through compute resize but recycles idle ones. No user-visible effect on healthy clients. 3. Confirm no migration job running (az containerapp job execution list -g ftl-prd-rg-cin -n ftl-prd-migrate).

CLIPortal

AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server update \
  -g ftl-prd-rg-cin -n ftl-prd-pg-cin \
  --sku-name Standard_D2ds_v5 \
  --tier GeneralPurpose \
  -o json

Portal → ftl-prd-pg-cin → Compute and storage.
Compute tier: change from Burstable to General Purpose.
Compute size: select Standard_D2ds_v5 (2 vCores).
Storage: leave at 64 GiB and autoGrow ON (set in Step 2).
Click Save at the bottom.
Confirmation dialog warns of brief failover. Click Continue.
Top bar shows "Updating server". Wait 5–10 min until "Running" with new SKU.

Risk: During the resize, PG performs a vertical SLB swap — there is a brief failover (~30–60 s) where new connections are rejected. Existing connections may also drop if they're idle. Mitigation: App layer (go-redis, pgxpool) reconnects automatically with backoff. Run this when no live match → max user impact = trade-window button briefly returns 503. If a match is live, postpone or coordinate with operator messaging.

Rollback: Same command with --sku-name Standard_B2ms --tier Burstable. Online, same ~5–10 min window.

Cost impact: +₹100/day (₹577/day vs ₹450/day at B2ms — both blended with idle). Cost report §13.4 already accounted for this in the recommended 5K/3K projection.

Post-action:

AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server show \
  -g ftl-prd-rg-cin -n ftl-prd-pg-cin \
  --query '{sku:sku.name, tier:sku.tier, storage:storage.storageSizeGb, autoGrow:storage.autoGrow}' -o json

Expected: { sku: "Standard_D2ds_v5", tier: "GeneralPurpose", storage: 64, autoGrow: "Enabled" }.

3.5 Step 5 — Upgrade Redis `Basic C0` → `Standard C1`¶

Basic C0 = 250 MB, no SLA, no replicas. Standard C1 = 1 GB, 99.9 % SLA, master+replica. Required for memory headroom + failover during the tournament.

Pre-action: Confirm Step 1 backup completed. Redis content is throwaway cache (rehydrated from PG); failover during resize loses in-flight pub/sub messages — acceptable because the dirty:* sets are PG-fed (idempotent on replay).

CLIPortal

AZURE_CONFIG_DIR=~/.azure-ftl az redis update \
  -g ftl-prd-rg-cin -n ftl-prd-redis-cin \
  --sku Standard --vm-size C1 \
  --set redisConfiguration.maxmemory-policy=allkeys-lru \
  -o json

Portal → ftl-prd-redis-cin → Settings → Scale.
Pricing tier: select Standard C1 (1 GB).
Click Select.
Portal shows "Scaling..." for ~5–15 minutes.
Settings → Advanced settings → set maxmemory-policy to allkeys-lru (LRU eviction when full). Click Save.

Risk: Redis resize triggers a ~30 s failover. App layer reconnects automatically; but during the failover window any user trade attempt will fail with redis: connection closed. The Go retry on Redis ops is bounded — if the failover takes longer than the retry budget, a few trades will return 5xx to the user. Mitigation: Schedule outside game time. Operator-side: monitor ftl-prd-api logs during the 30 s for ERROR redis lines; if more than 5 in the window, manually restart the api pod (az containerapp revision restart).

Rollback: Same command with --sku Basic --vm-size C0. Same failover risk.

Cost impact: +₹110/day. Cost report §13.4 already includes this.

Post-action:

AZURE_CONFIG_DIR=~/.azure-ftl az redis show \
  -g ftl-prd-rg-cin -n ftl-prd-redis-cin \
  --query '{sku:sku.name, capacity:sku.capacity, family:sku.family, replicas:replicasPerMaster}' -o json

Expected: { sku: "Standard", capacity: 1, family: "C", replicas: 1 }.

3.6 Step 6 — Enable required reviewers on `production` GitHub Environment¶

Currently both ftl-backend/production and ftl-frontend/production have NO required reviewers. Any push to release/prod auto-deploys. Per github-env-audit-2026-06-11.md this is a critical gap.

CLIPortal (GitHub UI)

# ftl-backend
PAT=$(git remote get-url origin | grep -oE "ghp_[A-Za-z0-9_]+")
GH_TOKEN="$PAT" gh api \
  --method PUT \
  "repos/JetaFutures/ftl-backend/environments/production" \
  -F 'reviewers[][type]=User' -F 'reviewers[][id]=<your-user-id>' \
  -F 'wait_timer=0'

# ftl-frontend
GH_TOKEN="$PAT" gh api \
  --method PUT \
  "repos/JetaFutures/ftl-frontend/environments/production" \
  -F 'reviewers[][type]=User' -F 'reviewers[][id]=<your-user-id>' \
  -F 'wait_timer=0'

To find your user ID:

GH_TOKEN="$PAT" gh api user --jq '.id'

For each of JetaFutures/ftl-backend and JetaFutures/ftl-frontend:

https://github.com/JetaFutures/ftl-{backend,frontend}/settings/environments.
Click production.
Tick Required reviewers → add @amalkrsihna (and a second operator if available).
Wait timer: 0 minutes (manual gate, not time-delay).
Deployment branches → restrict to release/prod only.
Click Save protection rules.

Risk: Wrong reviewer set → cannot ship. Forgetting the gate means uncontrolled prod deploys. Mitigation: Add at least two reviewers. Document in this runbook who they are: @amalkrsihna + (second operator TBD).

Rollback: Remove the reviewer requirement same way (UI: untick the box).

Post-action: Test by pushing a no-op commit to release/prod and confirming the deploy run pauses on the production environment job with a "Waiting for approval" gate.

3.7 Step 7 — Roll the new image to prod¶

This is its own runbook with the full branching strategy. See prod-deployment-runbook.md.

4. Post-launch verification (after Step 7)¶

Run the verification commands from prod-snapshot-2026-06-11.md §7. Confirm every row in §6 of the snapshot is now ticked off:

Snapshot row	Post-launch expected
Image SHA on all 3 services	new SHA (today's `release/staging` HEAD)
`APPLICATIONINSIGHTS_CONNECTION_STRING`	secretref `appinsights-conn`
`OTEL_TRACE_SAMPLE_RATIO`	`0.01`
`SPORTMONKS_LEAGUE_IDS` (plural)	`732`
`SPORTMONKS_LEAGUE_ID` (singular legacy)	removed
PG SKU	`Standard_D2ds_v5`
PG autoGrow	`Enabled`
Redis SKU + capacity	`Standard C1 (1 GB)`
LAW dailyQuotaGb	`30` (was set 2026-06-11 06:03 UTC)
GitHub `production` env reviewers	at least one user

5. Total expected duration¶

Step	Duration
1. Backup	1–3 min
2. autoGrow	<1 min
3. Env vars (3 apps, staggered)	3–5 min total
4. PG resize	5–10 min
5. Redis resize	5–15 min
6. GH reviewer gate	2 min
7. Prod deploy (see other runbook)	10–20 min
TOTAL	~30–60 min including waits

Pick a one-hour window outside any WC26 fixture. The first WC26 match kicks off 2026-06-11 19:00 UTC; aim for completion at least 90 min before.

6. If a step fails — escape ladder¶

A step failed?
├─ Step 1 (backup) failed
│   → Read az error, fix permission / quota issue, retry. Do NOT proceed to Step 2.
├─ Step 2 (autoGrow) failed
│   → autoGrow toggle is idempotent. Retry. If still failing, file Azure support
│     ticket; safe to proceed manually (operator monitors disk).
├─ Step 3 (env vars) failed for one app
│   → The other two apps already rolled. Retry for the failing app only.
│     Do not proceed to Step 4 if any app has wrong env values.
├─ Step 4 (PG resize) stalled >30 min
│   → Open Azure support ticket. Online resize CAN take longer if PG has
│     heavy in-flight work. Confirm via Metrics blade. Do not cancel mid-flight.
├─ Step 5 (Redis resize) failed
│   → Roll back to Basic C0 if Standard C1 cannot complete. Redis re-hydrates
│     from PG on connect; no data loss.
├─ Step 6 (reviewers) failed
│   → API requires user ID, not username. Re-fetch with `gh api user --jq '.id'`.
└─ Step 7 (deploy) failed
    → See prod-deployment-runbook.md §"Rollback".

7. Risks NOT mitigated by this runbook¶

WC26 fixture starts during the runbook. Mitigation: operator chooses window.
Azure regional outage. Single-zone deployment by design (HA=Disabled per cost report). Risk accepted.
Sportmonks API key exhaustion. Out of scope for this runbook; tracked in pre-launch checklist (CLAUDE.md §"Pre-Launch Security Checklist").
DDoS / credential stuffing. DISABLE_RATE_LIMIT=0 must be verified post-deploy (CLAUDE.md §"Pre-Launch Security Checklist" #7).

Prod Launch Runbook¶

0. Context¶

1. Pre-flight (do these ONCE before starting)¶

2. Step ordering — first things first¶

3. The steps¶

3.1 Step 1 — Backup Postgres + Redis BEFORE anything¶

3.2 Step 2 — Enable PG autoGrow¶

3.3 Step 3 — Set Container Apps env vars¶

3.4 Step 4 — Upgrade Postgres SKU B2ms → D2ds_v5¶

3.5 Step 5 — Upgrade Redis Basic C0 → Standard C1¶

3.6 Step 6 — Enable required reviewers on production GitHub Environment¶

3.7 Step 7 — Roll the new image to prod¶

4. Post-launch verification (after Step 7)¶

5. Total expected duration¶

6. If a step fails — escape ladder¶

7. Risks NOT mitigated by this runbook¶

3.4 Step 4 — Upgrade Postgres SKU `B2ms` → `D2ds_v5`¶

3.5 Step 5 — Upgrade Redis `Basic C0` → `Standard C1`¶

3.6 Step 6 — Enable required reviewers on `production` GitHub Environment¶