Skip to content

title: Production Cutover — Impact Analysis (2026-06-11 snapshot) status: reference last-updated: 2026-06-11 owners: [@amalkrsihna] audience: operator before any release/staging → release/prod promote

Production Cutover — Impact Analysis

Purpose. Read this BEFORE merging release/staging into release/prod. It captures the current state of prod, the gap between prod and staging, what will change for existing users when the gap is closed, and how to inspect or restore the prod database. The analysis itself is read-only; nothing in this doc was applied against production.

This document is a snapshot taken on 2026-06-11. Prod, staging, and the migration graph may have moved on since — re-run the inspection commands in §1 and §6 before treating any number here as current.

1. Prod current state (confirmed read-only at 2026-06-11)

Field Value
Active api revision ftl-prd-api--0000008
Active since 2026-05-22T15:07:17Z (≈ 20 days ago)
Image SHA fea5dc4 (= ftl-backend@fea5dc417e31f64b8591e4f4fbcd6ad074cf4325)
Highest migration in that build 000030_phone_unique_index
Replicas 1
Container health Healthy
PG SKU Standard_B2ms
PG storage 64 GB
Backup retention 7 days
Backup redundancy LRS (single region)
High availability Disabled (no read replica)
Earliest restore point at snapshot time 2026-06-04T16:06:35Z

Re-run these checks to refresh:

AZURE_CONFIG_DIR=~/.azure-ftl az containerapp revision list \
  --name ftl-prd-api --resource-group ftl-prd-rg-cin \
  --query "[?properties.active].{rev:name, image:properties.template.containers[0].image, created:properties.createdTime}" -o table

AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server show \
  --name ftl-prd-pg-cin --resource-group ftl-prd-rg-cin \
  --query "{retention:backup.backupRetentionDays, earliest:backup.earliestRestoreDate, geo:backup.geoRedundantBackup, ha:highAvailability.mode, sku:sku.name, storageGB:storage.storageSizeGb}" -o json

The PG firewall currently allows two entries: OperatorMachine (49.47.199.187/32) and AllowAzureServices (0.0.0.0/0 within Azure). Any direct psql or pg_dump must run from the operator machine; other machines need a temporary firewall rule (a write op).

2. Prod vs staging migration gap (13 migrations)

Staging is at migration 000043 after the 2026-06-11 cutover-prep work. Prod is at 000030. The 13 unmerged migrations are the entire CFD trading rewrite:

000031 create_cfd_positions                 — adds the CFD positions table
000032 create_margin_events                 — adds margin events
000033 add_position_imbalance_and_kmod      — adds k_mod + net_position_imbalance
000034 add_instrument_tier                  — adds tier classification
000035 create_position_outbox               — adds CFD position outbox
000036 add_tutorial_completed               — adds users.tutorial_completed
000037 add_cfd_positions_price_checks       — adds price-validity check constraints
000038 add_permanent_impact                 — adds instruments.permanent_impact
000039 bump_k_mod_default                   — k_mod default 0.01 → 0.05
000040 normalize_and_extend_event_bumps     — admin event-bump dictionary refresh
000041 add_persistent_idx                   — adds persistent index (had dirty-state issue on staging)
000042 repair_persistent_idx                — repaired the 000041 dirty state
000043 lower_k_mod_default                  — k_mod default 0.05 → 0.01 (ADR-0035)

Mechanically this is migrate up running 13 statements in order, each wrapped by golang-migrate in its own transaction. The migrate job runs BEFORE app rollout per deploy-prod.yml's needs: chain.

3. What changes for the ~100 existing prod users

3.1 Schema additions (additive, no data loss)

  • New tables created empty: cfd_positions, margin_events, position_outbox.
  • New columns added with defaults to existing rows:
  • instruments.k_mod (NUMERIC, default 0.05 after mig 000033; lands at 0.01 after mig 000043).
  • instruments.net_position_imbalance (INT, default 0).
  • instruments.permanent_impact (NUMERIC, default 0).
  • instruments.tier (TEXT, default 'C').
  • users.tutorial_completed (BOOL, default false).
  • Legacy tables untouched:
  • positions (mig 000004) stays. Existing legacy open positions remain valid in legacy format.
  • trades table semantics unchanged.

3.2 Trading behavior cut-over

  • Frontend + backend switch to CFD-only after the new image rolls. Memory: CFD-only trading; legacy /api/trade retiring (issue #68 in ftl-backend).
  • Open question that must be resolved BEFORE cutover: what happens to legacy positions held by the 100 existing users? Until issue #68's retirement plan is written, those positions are in an undefined post-cutover state. Read #68 before promoting.

3.3 Pricing

After mig 000033 runs, every instrument has k_mod defaulted (briefly 0.05, then 0.01 by mig 000043). Both net_position_imbalance and permanent_impact start at 0 because the cfd_positions table is freshly created and empty. So:

  • Existing current_price for each instrument (= live_base_price + k_mod * (0 + 0) = live_base_price) is unchanged at cutover.
  • First new CFD trade on prod moves price 0.01 × 5 = ₹0.05 per lot on a ₹100 instrument — same as staging now.
  • ADR-0017 timing risk (open CFD positions repricing at close) is effectively zero for the cutover itself because there are no open CFD positions to reprice. It only starts applying to positions opened after cutover, which is the new normal anyway.

3.4 Wallets, margin, washout

  • wallet_default_10000 (mig 000027) is already on prod. Wallet shape unchanged.
  • Margin / washout / equity rules (introduced by mig 000031–000032) become active. Users will see new UI cues (margin warning, washout messaging) that they didn't have before.

3.5 Event ceiling 10% → 30%

  • After cutover, cumulative event-driven micro_bump can climb to 30% of live_base_price (was 10%). Hot players during matches will see larger event-driven price moves than they did before. This is by design — the change re-balances the trade-vs-event channels.

3.6 Risk envelope summary

Risk Severity Mitigation
Migrate job fails mid-chain → schema_migrations.dirty=true medium (000041 caused this on staging) Re-run after clearing dirty flag; mig 000042 repair already in chain. PITR if needed.
Legacy positions in undefined state high Must resolve issue #68 retirement plan BEFORE cutover.
B2ms PG saturates during 13-mig chain medium Upgrade to D2ds_v5 BEFORE cutover (per memory wc26_preimpl_validation_2026-06-08).
Existing users see new UI elements without warning low In-app announcement before cutover.
Rollback window only 7 days medium Bump backup.backupRetentionDays to 14+ at least a week before cutover.

4. Sequence of events at cutover

When release/staging is fast-forwarded into release/prod:

  1. deploy-prod.yml fires on push and pauses at the production GitHub Environment approval gate. Manual approval required.
  2. On approval, four images build in parallel: api, ws, flusher, migrate.
  3. migrate Container App Job runs migrate up — 13 migrations execute in sequence, each in its own transaction.
  4. App rollout (az containerapp update) rolls three apps to new revisions. Old revisions drain.
  5. /health smoke check against the api FQDN.
  6. "Deployment complete" marker job.

Expected wall time on B2ms PG: ~5–10 minutes if all clean. The risky mid-job steps:

  • 000033 adds 3 NOT NULL columns with defaults to instruments (~5.7k rows). ALTER TABLE rewrites the heap. Seconds on B2ms, but the lock during the rewrite blocks reads/writes on instruments until done.
  • 000034 instrument tier: same shape, same lock window.
  • 000038 permanent_impact column: same.
  • 000041 / 000042 persistent_idx: this gave staging two attempts before going clean. Same risk on prod.

If any single migration fails mid-transaction, the migrate job marks schema_migrations.dirty=true for the failing version and exits non-zero. The migrate-job container app exits before app rollout starts (because deploy-prod.yml has needs: migrate on the rollout steps).

5. Backup posture for prod

5.1 What Azure gives you automatically

  • 7-day rolling retention with full daily snapshots.
  • PITR to any second within the retention window.
  • LRS (Locally Redundant Storage) — backups live in the same region as the server (Central India). No cross-region copy.

5.2 Listed snapshots at 2026-06-11

backup_639161859943042789
backup_639162724451611060
backup_639163589154008539
backup_639164453806593344
backup_639165317861760730
backup_639166180264955473
backup_639167044565144416

These names are auto-generated by Azure (the number is a .NET-ticks timestamp). One per day for the last 7 days. Azure Portal renders human-readable timestamps next to them.

5.3 Read-only inspection commands

# Inspect server-level backup config
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server show \
  --name ftl-prd-pg-cin --resource-group ftl-prd-rg-cin \
  --query "{retention:backup.backupRetentionDays, earliest:backup.earliestRestoreDate, geo:backup.geoRedundantBackup}" -o json

# List existing backups
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server backup list \
  --resource-group ftl-prd-rg-cin --name ftl-prd-pg-cin -o table

5.4 How to PULL a backup off-Azure

Azure Postgres Flex does NOT support direct download of its internal snapshots. The only off-Azure copy you can hold is a logical pg_dump produced from your operator machine (which is already on the prod PG firewall):

# Run from the operator machine at 49.47.199.187 (already in the firewall allow-list).
PRD_DSN=$(AZURE_CONFIG_DIR=~/.azure-ftl az keyvault secret show \
  --vault-name ftl-prd-kv-cin --name database-url --query value -o tsv)
TS=$(date -u +%Y%m%dT%H%M%SZ)
pg_dump --no-owner --no-acl --schema=public "$PRD_DSN" > ftl-prd-snapshot-$TS.sql
gzip ftl-prd-snapshot-$TS.sql

This dump captures schema + data at the moment it runs. Store it off the operator machine (e.g. Azure Blob in a separate sub, or another cloud) so it survives operator-machine loss.

5.5 Write ops needed at cutover (NOT executed by this analysis)

The following are write operations referenced for the cutover plan; they are explicitly not run here. Each must be operator-authorized at cutover time.

# Extra on-demand named backup before cutover (recommended).
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server backup create \
  --resource-group ftl-prd-rg-cin --name ftl-prd-pg-cin \
  --backup-name pre-cutover-$(date +%Y%m%d)

# Extend retention to 14 days at least a week before cutover (recommended).
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server update \
  --resource-group ftl-prd-rg-cin --name ftl-prd-pg-cin \
  --backup-retention 14

# Restore to a NEW server at a specific point in time (rollback path).
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server restore \
  --resource-group ftl-prd-rg-cin --name ftl-prd-pg-restored \
  --source-server ftl-prd-pg-cin \
  --restore-time "2026-06-10T23:00:00Z"

6. Pre-cutover checklist

These are pre-flight checks to run when the time comes. None of them are run by this document.

  • [ ] Resolve the legacy-position retirement plan in ftl-backend issue #68. Specifically: what happens to each of the ~100 users' existing legacy positions when CFD becomes the trading model? Backfill into cfd_positions? Auto-close at last legacy price? Freeze and grandfather? No cutover proceeds until that decision is recorded.
  • [ ] Bump prod PG backup retention to 14 days at least a week before cutover.
  • [ ] Upgrade prod PG SKU from Standard_B2ms to D2ds_v5 (memory wc26_preimpl_validation_2026-06-08). Schedule the upgrade for a no-traffic window.
  • [ ] Take a manual on-demand backup with a meaningful name (pre-cutover-YYYYMMDD).
  • [ ] Pull a logical pg_dump from the operator machine, store off-Azure.
  • [ ] Confirm staging has held green CI on release/staging for ≥ 48 hours.
  • [ ] Run the full 13-migration chain on a staging clone seeded from the prod dump (to verify the chain works against prod's exact data shape).
  • [ ] Schedule cutover for a no-match window (between matches, not during a fixture).
  • [ ] Prepare in-app announcement so the 100 users aren't surprised by margin/washout UI.
  • [ ] Identify the operator who will approve the production GitHub Environment gate.
  • [ ] Document the immediate rollback path: PITR restore to a point ≤ 30 min before the gate was approved.
  • ADR-0017 — Permanent price impact (introduces k_mod semantics).
  • ADR-0035 — Lower k_mod default + raise event cap (this is mig 000043).
  • Spec architecture/specs/amm-pricing.md — current pricing formula.
  • Spec architecture/specs/cfd-pricing-and-wallet.md — CFD model that prod will adopt at cutover.
  • Memory entry wc26_preimpl_validation_2026-06-08 — context on the prod PG SKU upgrade task.
  • Memory entry cfd_only_trading — points to issue #68 for legacy /api/trade retirement.
  • Memory entry azure_billing_reality — current prod RG burn rate (~₹650/day idle).