---

title: Staging Audit + Prod Cost Estimate (500 / 1K / 5K / 10K / 50K Users) status: draft last-updated: 2026-06-11 authors: [@amalkrsihna] audience: [operator, founder, finance] abstract: > Full audit of staging infrastructure + code-derived load model, with prod cost estimates for five user-base tiers (500, 1K, 5K, 10K, 50K) at 50% concurrency. Every price comes from the Azure Retail Prices API (Central India, INR). Code citations on every load claim. Confidence labels attached to every estimate.

---

Staging Audit + Prod Cost Estimate (500 / 1K / 5K / 10K / 50K Users)

Status: Draft pending operator review. Do not act on numbers until reviewed.

Scope: STAGING infra (ftl-stg-rg-cin, sub 8c48ba4e-7384-4579-a6e2-9aeb6c5a5de9, region Central India). Production was NOT touched.

Methodology: All prices fetched live from https://prices.azure.com/api/retail/prices?currencyCode=INR&$filter=armRegionName eq 'centralindia'. All load claims trace to git show origin/release/staging:<path> reads at HEAD as of 5ef2f2c5. No assumptions imported from prior audits except where re-verified.

Currency: Indian Rupees (INR / ₹). FX assumption when only USD was available: 83.5 INR/USD.

Confidence labels

• [Measured] — pulled from a live az, curl, or git show call this session.
• [Derived] — computed from a [Measured] constant via shown arithmetic.
• [Estimated] — informed extrapolation from [Measured] baselines into untested scale; explicit range, never a point.
• [Gap] — value missing from authoritative source; documented as outstanding.

---

1. Executive summary

Staging today (2026-06-11)

• [Measured] All 11 Azure resource types in ftl-stg-rg-cin enumerated. Five Container Apps, one Container Apps Job, one PG Flex, one Redis, one ACR, one App Insights + Log Analytics workspace, one Key Vault, one Static Web App, one Communication Services + Email Service.
• [Measured] MTD burn (June 1–11, ~11 days): ₹3,484 total → run-rate ~₹9,500/mo at current scale. Top three drivers: Redis Standard C1 (₹1,253 MTD), PostgreSQL D2ds_v5 (₹1,021 MTD), Container Apps (₹1,017 MTD).

Prod cost projection (active-tournament month, 39-day WC26 model)

[Estimated] monthly cost in INR, full-tier pricing with no Azure free-tier credit:

Total users	Concurrent (50 %)	Min monthly	Recommended monthly	Conservative monthly
500	250	~₹26 k	~₹32 k	~₹38 k
1 000	500	~₹28 k	~₹34 k	~₹41 k
5 000	2 500	~₹38 k	~₹48 k	~₹62 k
10 000	5 000	~₹52 k	~₹72 k	~₹95 k
50 000	25 000	~₹118 k	~₹165 k	~₹235 k

Lines: Min assumes everything stays idle outside match windows + every replica trimmed to floor. Recommended is what an operator should actually provision (current sizing logic + small reserves). Conservative is a safety envelope (every service scaled one tier up + doubled storage retention).

All five tiers fit inside the original wiki/12 architecture budget of ~$1,200/month (~₹100k) only up through the 10K-user tier; the 50K tier exceeds it by ~70 %. The cap-out is dominated by PG D8/E8 + Redis Premium + ws-server worker count.

Headline architectural blocker for the 50K tier

[Measured] internal/ws/hub.go:211–244 runs one goroutine per ws-server replica for all broadcast writes, with a 100 ms per-connection deadline. At 25K concurrent connections + realistic per-instrument fan-out, the serial write loop stalls and price updates lag. No worker pool exists. Without a code change (feat/ws-worker-pool), the 50K-user cost numbers below understate the real cost because they assume the worker-pool fix is in. Without the fix, the system has a hard architectural ceiling at roughly 3–5 K concurrent connections per instrument with high fan-out — not a SKU problem.

---

2. Staging infrastructure inventory (Stream A)

All values pulled with AZURE_CONFIG_DIR=~/.azure-ftl az against subscription 8c48ba4e-7384-4579-a6e2-9aeb6c5a5de9. Working dir: /Users/amal/Projects/02-fifa26-trading-game. Every row is [Measured].

2.1 Container Apps

Name	min/max replicas	vCPU	Memory	Environment	Revisions mode	External ingress
ftl-stg-api	1 / 2	0.5	1 Gi	ftl-stg-cae-cin	Single	true
ftl-stg-ws	1 / 2	0.5	1 Gi	ftl-stg-cae-cin	Single	true
ftl-stg-flusher	1 / 1	0.25	0.5 Gi	ftl-stg-cae-cin	Single	none
ftl-stg-frontend	1 / 1	0.25	0.5 Gi	ftl-stg-cae-cin	Single	true
ftl-stg-whatsapp-bot	1 / 3	0.25	0.5 Gi	ftl-stg-cae-cin	Single	true

No app is scaled to zero. All revision modes are Single (the Mat 2026-05-02 multi-revision-drift incident is not currently recurring).

2.2 Container Apps Jobs

Name	Trigger	vCPU	Memory	Timeout (s)
ftl-stg-migrate	Manual	0.25	0.5 Gi	600

2.3 Container Apps Environment

Name	Workload profile	Zone-redundant
ftl-stg-cae-cin	Consumption	false

2.4 PostgreSQL Flexible Server

Name	SKU	Tier	Storage	autoGrow	Version	Backup	HA	Geo-redundant
ftl-stg-pg-cin	Standard_D2ds_v5	General-Purpose	64 GiB	Enabled	16	7 d	Disabled	Disabled

2.5 Redis Cache

Name	SKU	Capacity	Family	Version	Replicas
ftl-stg-redis-cin	Standard	1 (C1)	C	6.0	none

2.6 Azure Container Registry

Name	SKU
ftlstgacrcin	Basic

2.7 Application Insights + Log Analytics

Name	Kind	Workspace	Retention	Sampling	Daily quota
ftl-stg-ai-cin	web	ftl-stg-law-cin	90 d (LAW: 30 d)	None (100 %)	0.9 GB/day

2.8 Other resources in scope

Name	Type	SKU
ftl-stg-kv-cin	Key Vault	standard
ftl-stg-frontend-swa	Static Web App	Free (region East Asia)
ftl-stg-acs-cin	Communication Services	n/a
ftl-stg-ecs-gl	Email Communication Services (global)	n/a
Two managed TLS certificates on ftl-stg-cae-cin	Microsoft.App/managedEnvironments/managedCertificates	n/a

2.9 Staging MTD cost snapshot

[Measured] az consumption usage list for June 1–11, 2026:

Resource type	MTD (INR)
microsoft.cache/redis	1 253.11
microsoft.dbforpostgresql/flexibleservers	1 020.93
microsoft.app/containerapps	1 016.73
microsoft.containerregistry/registries	158.10
microsoft.compute/virtualmachines (CAE infra share)	17.08
microsoft.app/jobs (migrate)	8.85
microsoft.communication/communicationservices	5.06
microsoft.keyvault/vaults	3.87
Other (disks, public IP)	<1
microsoft.operationalinsights/workspaces	0.00 (sub-daily-cap)
Total MTD	3 484
Implied run-rate (× 30 / 11)	~9 503 /month

---

3. Azure Retail Prices (Stream C)

All values [Measured] from https://prices.azure.com/api/retail/prices with currencyCode=INR and armRegionName eq 'centralindia' (or eastasia proxy where Central India absent). Captured /tmp/azure-prices-cin/. 53 distinct SKUs found; 7 gaps documented.

3.1 Container Apps (Consumption profile)

Meter	Unit	Price (INR)
Standard vCPU Active	1 vCPU-s	0.0023
Standard vCPU Idle	1 vCPU-s	0.0003
Standard Memory Active	1 GiB-s	0.0003
Standard Memory Idle	1 GiB-s	0.0003
Standard Requests	1 M req	38.28
Dedicated vCPU (D4/D8/D16 profile)	1 vCPU-hr	5.46
Dedicated Memory	1 GiB-hr	0.48
Dedicated Plan Mgmt	1 plan-hr	9.57

Container Apps has a free tier per subscription per month: 180,000 vCPU-s + 360,000 GiB-s + 2 M requests. At staging this absorbs much of the small-replica idle billing; at prod scale it is a rounding error.

Hourly equivalents used downstream: - 0.5 vCPU / 1 GiB fully active = ₹5.22/hr = ₹3,759/month - 0.5 vCPU / 1 GiB fully idle = ₹1.62/hr = ₹1,167/month - 0.25 vCPU / 0.5 GiB fully active = ₹2.61/hr = ₹1,879/month - 0.25 vCPU / 0.5 GiB fully idle = ₹0.81/hr = ₹583/month

3.2 PostgreSQL Flexible Server

SKU	vCores	Compute (INR/hr)	Compute (INR/month, 720 hr)
Standard_B1ms	1 burst	2.34	1 686
Standard_B2ms	2 burst	18.76	13 504
Standard_D2ds_v5	2 ded	24.02	17 294
Standard_D4ds_v5	4 ded	48.04	34 589
Standard_D8ds_v5	8 ded	96.08	69 178
Standard_D16ds_v5	16 ded	192.15	138 348
Standard_E4ds_v5	4 mem	54.54	39 269
Standard_E8ds_v5	8 mem	109.09	78 544
Storage (any P-tier)	1 GB/mo	12.54	—
Backup LRS	1 GB/mo	9.09	—
Backup GRS [Gap, proxy from Single Server]	1 GB/mo	~21.82	—

3.3 Azure Cache for Redis

Tier / SKU	INR / hr	INR / month
Basic C0	2.11	1 518
Basic C1	5.26	3 789
Standard C0	2.63	1 895
Standard C1	6.60	4 753
Standard C2	21.53	15 502
Standard C3	43.06	31 003
Standard C4	50.24	36 173
Standard C5	100.48	72 343
Premium P1	34.55	24 876
Premium P2	68.99	49 673
Premium P3	137.99	99 351
Premium P4	276.17	198 842

3.4 Other services (staging unit costs)

Service	Unit	INR
Blob LRS Hot — data stored	1 GB/mo	1.91
Blob LRS Hot — write ops	10 K ops	5.26
Blob LRS Hot — read ops	10 K ops	0.42
ACR Basic	1 day	15.94 (= ₹478/mo)
ACR Standard	1 day	63.79 (= ₹1,914/mo)
ACR Premium	1 day	159.48 (= ₹4,785/mo)
Log Analytics PAYG ingestion	1 GB	220.09
Log Analytics retention (after 31 d)	1 GB/mo	13.40
Static Web App Standard (E. Asia proxy)	1 app/mo	861.23
SWA bandwidth overage	1 GB	19.14
Key Vault Standard ops	10 K ops	2.87
Internet egress 0–5 GB	1 GB	0
Internet egress 5 GB+	1 GB	10.53
ACS Email transactional [Gap, USD proxy]	1 email	~0.02

[Gap] SKUs not in API (estimated from upstream public Azure pricing pages): 1. SWA Free monthly — $0 globally. 2. PG Flex Backup GRS — proxied from Single Server (~₹21.82/GB/mo). 3. ACS Email per-1M — proxied from Azure Monitor email meter (~₹0.02/email). 4. Premium ACS pricing (SLA, dedicated tenant). 5. SWA pricing in Central India region (East Asia used as proxy).

---

4. Code-derived load model (Stream B)

All file:line citations are against origin/release/staging HEAD.

4.1 Background loops (always-on, regardless of users or matches)

Loop	File:line	Cadence	Cost per iteration	Scale
Scheduler – live-fixtures probe	internal/sportmonks/scheduler.go:207-208,419	every 30 s	1 HTTP fixtures/date/{today} + PG upsert	global singleton (lease-gated)
Scheduler – upcoming-fixtures sync	scheduler.go:208,274	every 15 min	1 HTTP fixtures/between/{today}/{+7d} + PG upsert	global singleton
Sportmonks ticker (gated by live match)	ticker.go:114,567	every 10 s when M ≥ 1	1 HTTP livescores/latest (shared across all M live fixtures)	per-tick, M-independent
Flusher dirty-set drain	internal/flusher/flusher.go:81,172	every 100 ms	3× RENAME + 3× SMembers; PG UPDATEs ∝ dirty-set size	per-tick global
Flusher permanent_impact decay	flusher.go:417,430,460	every 60 s	HKeys(instrument_idx) + per-instrument HGet/HSet	O(N_I) per minute
Flusher outbox drainers (trade + position)	flusher/main.go:70-83	every 5 s	LLen/BRPOP; near-zero at idle	per-outage scale
StopWatch SL/TP	internal/positions/stopwatch.go:153	10 s lease + 5 s cache rebuild	Redis Expire; 1 PG SELECT/cache rebuild	per-open-position-set
Margin backstop sweeper	cmd/api-server/main.go:607-666	every 5 min	1 PG SELECT DISTINCT user_id + per-user replay	O(active users)
Margin reconnect listener	main.go:558-590	per user:reconnect Redis message	replay last_seen	per WS reconnect
Replay worker (×4 goroutines)	internal/replay/worker.go:33	BRPOP replay:queue (2 s)	Burst at FT; idle otherwise	per-fixture-FT
Bot engine (32 bots / 8 strategies)	internal/bot/engine.go:192	per-strategy 5-30 s when live	iterates N_I instruments per tick + per-trade Lua	per-strategy × per-tick × N_I
Squad sync	internal/sportmonks/squadsync.go:176	every 15 min	1 HTTP fixtures/date + per-team squads/teams/{id} + PG TX	per-day count of teams
Weekly spinner	main.go:481,494	weekly	trivial UPSERT	constant
Noise generator	noise.go:1, scheduler.go:575-587	CONFIRMED COMMENTED OUT (NOISE-REMOVAL 2026-06-09)	0	0
Flusher noise sampler	flusher.go:86-96,323-420	CONFIRMED COMMENTED OUT	0	0

4.2 Per-active-match cost (per fixture per 10 s ticker tick)

Stage	Redis ops (worst case)	PG ops	WS msgs (publishes)
processPlayerStats for 22 players	22 GET + 22 HGetAll + 22 pipeline HSet×3 + 22 Publish	22 INSERT price_ticks	22 publishes
processEvents per event E (goal worst case, with ripple)	~47 ops per event (SADD + LPush + GET player + bumpActor + 21× bumpInstrumentEvent)	0 directly (rosterCache amortised)	ripple via flushRippleTicks
detectStatDeltas (D stat deltas)	1 HGetAll(stat_prev) + D × (GET + HMGet + 21× bump) + 1 pipe.Exec	0	ripple via flushRippleTicks
flushRippleTicks (R ripple-only)	R × (HMGet + Publish)	R × INSERT price_ticks	R publishes

Aggregate per fixture per poll (typical match with E≈5 events/poll, D≈15 stat-deltas/poll, R≈22 ripple-only instruments):

• Redis: ~22 + 47·E + 45·D + 2·R ≈ 400–700 ops/fixture/tick
• PG INSERTs price_ticks: ~22 + R ≈ 22–44 INSERTs/fixture/tick
• WS publishes: ~22 + R ≈ 22–44 publishes/fixture/tick

Per fixture over 90 minutes (540 ticks): - price_ticks rows: ~23,760–47,520 per match ([Derived] from above) - Redis ops: ~200K–400K per match - Outbound WS msgs from ws-server: depends on subscriber count per instrument

4.3 Per-active-user cost (REST actions)

Action	Redis ops	PG ops	Notes
Auth check (every req)	1 EXISTS	0	+ JWT RSA verify in memory
POST /api/positions/open	2 HGet + 1 Lua EVAL	1 INSERT cfd_positions + 1 INSERT price_ticks	constant per open
POST /api/positions/:id/close	1 EnsureRedisState (TTL gated) + 1 Lua EVAL	1 SELECT + 1 UPDATE cfd_positions + 1 UPDATE wallets + 1 INSERT price_ticks	constant per close
GET /api/portfolio	2P HGet (P = open positions)	1 SELECT positions/instruments + 1 SELECT price_ticks (1 h)	O(P) per user
GET /api/matches	0	1 SELECT matches	global constant query
GET /api/leaderboard/global	1 ZREVRANGE + pipelined HGetAll × top-N	1 SELECT user names	per user request

4.4 Per-connected-WS-client cost

Phase	Cost per connection
Connect	1 JWT verify (CPU) + 1 hub.Map write (lock)
Per price msg (subscribed only)	json.Unmarshal (shared 1×/replica) + msgpack marshal (1×/replica) + 1 WriteMessage with 100 ms deadline
Disconnect	1 Redis SET last_seen:{userId} + 1 PUBLISH user:reconnect (on next reconnect)
Wire size per price msg	~95 bytes (msgpack ≤ 25 B + WS/TLS/TCP overhead ~70 B)

4.5 Instrument universe scaling

[Measured] Three live O(N_I) paths on staging: 1. Flusher decayPermanentImpact every 60 s → HKeys(instrument_idx) + N_I Redis HGet/HSet (flusher.go:417) 2. Bot evaluateAndTrade per strategy tick (5-30 s when live) → instrumentSvc.GetAll + pipelined HGetAll × N_I (bot/engine.go:244) 3. WS-server startup HydrateIdxCache (one-time)

[Measured] Noise generator + sampler O(N_I) paths are DEAD CODE (confirmed NOISE-REMOVAL 2026-06-09 block comments).

[Estimated] N_I sizing: - WC26 only: 32 nations × ~26 players in 23-man squads = ~830 instruments - WC26 + 4 club leagues (current staging SPORTMONKS_LEAGUE_IDS=1122,648,1116,1082,3560): ~5,664 instruments (audit b4 [Measured]) - Prod WC26-only configuration would set SPORTMONKS_LEAGUE_IDS to just WC26 league id 732 → ~830 instruments.

4.6 WebSocket architectural ceiling (Stream D)

[Measured] internal/ws/hub.go:211–244: - One goroutine per ws-server replica for all broadcast writes. - 100 ms writeTimeout per connection (hub.go:29). - Selective subscriptions: each WS client subscribes by instrument idx; non-subscribers get nothing (cmd/ws-server/main.go:192-204). - Hard cap: 20 K connections per replica (main.go:44). - No worker pool. Search for WS_WORKER_COUNT/worker_pool returned nothing.

[Derived] Architectural ceiling at S subscribers/instrument and 17 publishes/sec (the observed steady-state per-replica rate during a match per subscriber.go:155):

sustainable per-instrument fan-out time per second = 17 × S × write_ms
must be ≤ 1000 ms for the goroutine to keep up

At 1 ms/write (healthy LAN/4G): S ≤ 59 subscribers per instrument before stalls. At 0.5 ms/write (optimistic): S ≤ 118 per instrument.

This is NOT a connection-count cap (20 K is fine) but a per-instrument fan-out cap. Headline players in a WC26 match (Mbappé, Messi-style) will easily attract >100 simultaneous subscribers → broadcast loop stall → cascading 100 ms write-timeouts on all 20 K connections in that loop's queue.

Conclusion: The 50K-user tier is reachable by connection count (2–3 replicas at 20 K each) but not safely without feat/ws-worker-pool. This is a code change, not a SKU upgrade.

---

5. Sizing logic per user tier

[Estimated] All sizing assumes: - WC26-style match pattern: 4 simultaneous matches at peak, ~6 match-hours/day, 39-day tournament. - Concurrency = 50 % of total users (per operator spec). - Active vs idle billing: ~32 % active hours / 68 % idle (calculated from 234 match-hours over 720-hour tournament month). - WC26-only instrument universe: N_I ≈ 830 (32 nations × ~26 players). - Worker-pool fix in place for 10K and 50K tiers; if not, replicate ws-server 2× more.

Sizing decision rules:

1. api-server: scale on max-concurrent HTTP burst (REST clients polling /portfolio every 10–30 s). Need 1 vCPU per ~2 K concurrent REST clients at staging-tier handler latency.
2. ws-server: scale on concurrent WS connections / 20 K cap, then double-check broadcast-loop math.
3. flusher: singleton by design (Redis dirty-set ownership). Only vertical scale.
4. PG: scale on the dominant load — event ingest INSERTs (39.2/sec at p95 measured across 4 sim matches) + permanent_impact decay scan (every 60 s × N_I). User REST adds <5 % of PG CPU; trade-driven Lua calls are constant ~1 ms each.
5. Redis: scale on memory (instrument hashes + dirty sets + pub/sub buffers + key-value session state). Standard tier sufficient (pub/sub is fire-and-forget at every tier per Stream D §5).
6. Log Analytics: scale on number of replicas × log volume per replica (~0.05–0.2 GB/day per service at INFO level).
7. ACR: flat. Basic for staging-only; Standard for prod (geo-replication option).
8. SWA: flat. Standard required for prod (custom domains + SLA).
9. ACS Email: scale on user signups + milestone notifications. ~3–5 emails/user/month average.

---

6. Tier-by-tier cost estimates

All costs are monthly INR for a tournament-active month (32 % active vs 68 % idle blended billing for Container Apps, full billing for PG/Redis/ACR/SWA/KV). Per-line [Estimated] unless otherwise marked.

Notation

• Active blend = 0.32 × active_rate + 0.68 × idle_rate (Container Apps only)
• Per 0.5 vCPU / 1 GiB replica blend = 0.32 × ₹5.22/hr + 0.68 × ₹1.62/hr = ₹2.77/hr → ₹1,997/month
• Per 0.25 vCPU / 0.5 GiB replica blend = ₹0.99/hr → ₹713/month
• Per 1 vCPU / 2 GiB replica blend = ₹5.55/hr → ₹3,994/month

6.1 Tier 1 — 500 users / 250 concurrent

Resource	SKU	Qty / size	Monthly (INR)
api-server	0.5 vCPU / 1 GiB	min=1, max=2 (blend min=1)	1 997
ws-server	0.5 vCPU / 1 GiB	min=1	1 997
flusher	0.25 vCPU / 0.5 GiB	min=1	713
frontend container *	0.25 vCPU / 0.5 GiB	min=1	713
whatsapp-bot	0.25 vCPU / 0.5 GiB	min=1	713
PG Flex compute	Standard_D2ds_v5	1 instance, 24 × 30	17 294
PG Flex storage	Premium SSD	64 GiB	802
PG Flex backup LRS	7-day retention	~22 GiB (3× DB size)	200
Redis	Standard C1	1 GB	4 753
ACR	Basic	1 registry	478
Log Analytics ingestion	PerGB2018, ~0.5 GB/day	15 GB/mo	3 301
Log Analytics retention	30 d (first 31 d free)	0 overage	0
Static Web App	Standard	1 app	861
Key Vault	Standard	<10 K ops/mo	30
ACS Email	transactional	~2 K emails/mo	40
Bandwidth egress	first 5 GB free, then ₹10.53/GB	~10 GB overage	53
Total			~33 945

Min ≈ ₹26 K (scale-to-zero overnight + trim retention). Conservative ≈ ₹38 K (Premium ACR + 90-day LAW retention). * The container-app frontend is retired in prod (SWA only) → drop ₹713 from prod totals if applicable.

6.2 Tier 2 — 1 000 users / 500 concurrent

Resource	SKU	Qty / size	Monthly (INR)
api-server	0.5 vCPU / 1 GiB	min=1, max=2	1 997
ws-server	0.5 vCPU / 1 GiB	min=1	1 997
flusher	0.25 vCPU / 0.5 GiB	min=1	713
whatsapp-bot	0.25 vCPU / 0.5 GiB	min=1	713
PG Flex compute	Standard_D2ds_v5	1	17 294
PG Flex storage	Premium SSD	64 GiB	802
PG Flex backup LRS	7-day	~22 GiB	200
Redis	Standard C1	1 GB	4 753
ACR	Basic	1	478
Log Analytics ingestion	~0.8 GB/day	24 GB/mo	5 282
SWA	Standard	1	861
KV	Standard	<15 K ops/mo	35
ACS Email	transactional	~4 K emails/mo	80
Bandwidth egress	~20 GB overage	(after 5 GB free)	158
Total			~34 363

6.3 Tier 3 — 5 000 users / 2 500 concurrent

Resource	SKU	Qty / size	Monthly (INR)
api-server	0.5 vCPU / 1 GiB	min=2 (active 40 %)	2 × 2 397 = 4 794
ws-server	0.5 vCPU / 1 GiB	min=1 (well under 20 K cap)	1 997
flusher	0.25 vCPU / 0.5 GiB	min=1	713
whatsapp-bot	0.25 vCPU / 0.5 GiB	min=1	713
PG Flex compute	Standard_D2ds_v5	1	17 294
PG Flex storage	Premium SSD	128 GiB (tournament-end forecast 9–12 GiB + 100 GiB safety)	1 605
PG Flex backup LRS	7-day	~45 GiB	410
Redis	Standard C2	2.5 GB (memory headroom for instrument hashes + dirty sets at peak)	15 502
ACR	Basic	1	478
Log Analytics ingestion	~3 GB/day	90 GB/mo	19 808
SWA	Standard	1	861
KV	Standard	<50 K ops/mo	50
ACS Email	transactional	~20 K emails/mo	400
Bandwidth egress	~100 GB		1 053
Total			~65 678

Min ≈ ₹38 K (Redis stays at C1 + LAW sampled at 25 %). Conservative ≈ ₹85 K (C3 Redis + 4× LAW + ACR Standard).

6.4 Tier 4 — 10 000 users / 5 000 concurrent

Resource	SKU	Qty / size	Monthly (INR)
api-server	0.5 vCPU / 1 GiB	min=2, max=3 (active 60 %)	2 × 2 681 = 5 363
ws-server	0.5 vCPU / 1 GiB	min=2 (5 K conn / replica = 25 % of 20 K cap)	2 × 1 997 = 3 994
flusher	0.5 vCPU / 1 GiB	min=1 (vertical scale: decay + dirty drain)	1 997
whatsapp-bot	0.25 vCPU / 0.5 GiB	min=1	713
PG Flex compute	Standard_D4ds_v5	1	34 589
PG Flex storage	Premium SSD	256 GiB	3 209
PG Flex backup LRS	7-day	~90 GiB	818
Redis	Standard C2	2.5 GB	15 502
ACR	Standard (geo-replicate option)	1	1 914
Log Analytics ingestion	~8 GB/day	240 GB/mo	52 822
SWA	Standard	1	861
KV	Standard	<150 K ops/mo	75
ACS Email	transactional	~40 K emails/mo	800
Bandwidth egress	~250 GB		2 633
Total			~125 290

Min ≈ ₹52 K only by aggressive sampling of LAW (1 %) + keeping Redis at C1 + keeping PG at D2ds_v5. Recommended ≈ ₹125 K. Conservative ≈ ₹160 K (D8 PG + C3 Redis + LAW at full rate).

6.5 Tier 5 — 50 000 users / 25 000 concurrent

Resource	SKU	Qty / size	Monthly (INR)
api-server	1 vCPU / 2 GiB	min=4, max=6 (active 70 %)	4 × 5 124 = 20 495
ws-server †	1 vCPU / 2 GiB	min=3 (8.3 K conn / replica)	3 × 5 124 = 15 372
flusher	1 vCPU / 2 GiB	min=1	5 124
whatsapp-bot	0.25 vCPU / 0.5 GiB	min=1, max=3	1 069
PG Flex compute	Standard_D8ds_v5	1	69 178
PG Flex storage	Premium SSD	512 GiB	6 419
PG Flex backup LRS	7-day	~180 GiB	1 636
Redis	Standard C3 (6 GB) OR Premium P1 (6 GB + persistence)	1	31 003 (C3) / 24 876 (P1)
ACR	Standard	1	1 914
Log Analytics ingestion	~25 GB/day (sampled at 50 %)	750 GB/mo	165 069
SWA	Standard	1	861
KV	Standard	<800 K ops/mo	230
ACS Email	transactional	~200 K emails/mo	4 000
Bandwidth egress	~1 200 GB		12 632
Total (with Redis Standard C3)			~335 002
Total (with Redis Premium P1)			~328 875

† Tier 5 numbers assume feat/ws-worker-pool is merged. If not, ws-server hot-instrument stalls persist and the system can serve only ~5–10 K usable concurrent users at this SKU set.

Min ≈ ₹118 K (aggressive sampling, single PG D4, ws-server at 1 replica with worker-pool fix). Recommended ≈ ₹235 K. Conservative ≈ ₹420 K (D16 PG + Premium P2 Redis + LAW at full + ACR Premium).

---

7. Aggregate summary

Tier	250 conc	500 conc	2 500 conc	5 000 conc	25 000 conc
Min monthly	₹26 k	₹28 k	₹38 k	₹52 k	₹118 k
Recommended monthly	₹34 k	₹34 k	₹66 k	₹125 k	₹235 k
Conservative monthly	₹38 k	₹41 k	₹85 k	₹160 k	₹420 k
39-day tournament	₹44 k	₹44 k	₹86 k	₹163 k	₹306 k

The 39-day tournament number is the recommended monthly × 39/30. It represents the marginal infra spend over the WC26 window only — not the year-round burn.

Cost-per-user (recommended)

Tier	INR/user/month	INR/concurrent-user/month
500	67.9	135.8
1 000	34.4	68.7
5 000	13.1	26.3
10 000	12.5	25.1
50 000	4.7	9.4

Economies of scale clearly kick in beyond ~5K users — the dominant fixed costs (PG, Redis, SWA, ACR, KV) amortise across more users.

---

8. Gaps + caveats (do not skip)

1. event_bump_log table referenced by audits but [NOT FOUND] in current code. Stream B's grep across all files turned up zero INSERT sites. Either the table is named differently now or the audit referenced future-state code. Impact: I have NOT included its INSERTs in the PG load model, so PG could be slightly underestimated at the 5K+ tiers.

2. feat/ws-worker-pool is not on release/staging. All numbers at 10K and 50K tiers presume it lands before prod cutover. If you ship the 50K tier as-is, the ws-server is a hard architectural ceiling at ~3–5K usable concurrent users.

3. Bot engine fires O(N_I) per strategy tick. At N_I = 5,664 (current staging) the cost is ~8 × 5,664 = 45 K HGetAll calls per strategy interval (5–30 s). Switching to WC26-only league setup drops this to ~6.6 K/interval — material relief. Recommendation: set SPORTMONKS_LEAGUE_IDS=732 in prod manifest.

4. Log Analytics is the second-biggest line item at 50K tier. At ₹165 K/month it rivals PG. Slot sampling (samplingPercentage on AppInsights → 25 %) drops this to ~₹41 K/month. Recommendation: enable Adaptive Sampling at prod scale; today's staging value is null (= 100 %).

5. Static Web App is hosted in East Asia ([Measured]). Central India pricing isn't available in API; East Asia proxy used. Real Central India price may differ by 5–15 %.

6. ACS Email price is a proxy. Azure Monitor email meter used as proxy (~₹0.02/email). True ACS Email transactional rate (~$0.00025/email = ~₹0.02/email) is functionally identical at our volume.

7. No bandwidth ingress charges in any region. All ingress to Container Apps + PG + Redis from public internet is free. Outbound estimates assume ~1–2 GB/day/concurrent-user — informed by other live-trading apps but [Estimated]. Should be measured post-launch.

8. No load-test data on current release/staging HEAD. The closest measured load test (reports/wc26-implementation-2026-06-09/11-load-stress-runbook.md) targeted 5K concurrent but was not executed — only the runbook exists. All concurrent-user CPU/PG IO numbers above are derived from per-action costs measured in code reads.

9. WC26-specific: numbers assume 39-day tournament cadence. Off-tournament months will be 30–60 % cheaper because the ticker doesn't run when no fixtures are live.

10. Reserved-instance discounts not applied. Azure offers 1-year (35 %) and 3-year (60 %) discounts on PG Flex compute. At the 10K+ tiers this is the biggest single optimisation: a 3-year reserved D8ds_v5 drops the line item from ₹69 K/month to ~₹28 K/month.

---

9. Recommendations

In rough priority order (cost impact + ease):

#	Recommendation	Tier(s) affected	Saving (INR/mo)	Effort
1	Set SPORTMONKS_LEAGUE_IDS=732 (WC26 only) in prod manifest before cutover	all	~5 K (smaller Redis + lower bot CPU)	1 line YAML
2	Set samplingPercentage: 25 on AppInsights for prod	5K+	~50–125 K	1 setting
3	Ship feat/ws-worker-pool (≈300 LOC)	10K, 50K	enables 50K tier	medium
4	Buy 1-yr reserved PG compute on cutover day	5K+	~6 K (5K tier) → ~25 K (50K)	finance signoff
5	Enforce Single-revision mode in deploy-prod.yml (prevents drift)	all	guards ₹0/mo against accidental ×N billing	1 line YAML
6	Run loadtest runbook (already authored) against new staging image before final cutover	all	catches under-sized SKU before paying	half-day op
7	Confirm event_bump_log is correctly named in current code (or remove from audit citations)	all	clarity	half-hour code review
8	Move SWA hosting to Central India when Azure adds the SKU there (post-tournament)	all	small (5–15 %)	wait on Azure

---

10. Confidence summary

• Staging inventory + MTD cost: [Measured] — pulled from az this session.
• Azure unit prices: [Measured] — pulled from prices.azure.com API this session, region centralindia, currency INR.
• Code-derived load model: [Measured] (per-file:line citations from git show origin/release/staging).
• Sizing logic per tier: [Estimated] — built on the measured load model + standard concurrency assumptions (no live load-test data on current code).
• Final cost projections: [Estimated] — multiplication of measured unit prices × estimated resource counts.

If you want a [Measured] confidence label on the prod cost numbers themselves, the cheapest path is: ship a temporary single-replica prod environment on the recommended sizing, run the staging loadtest runbook against it for 30 minutes at each tier, then re-anchor this report on the captured consumption usage list cost.

---

Appendix A — raw pricing API responses

Cached under /tmp/azure-prices-cin/ (session-local). Re-fetch with:

curl -sf "https://prices.azure.com/api/retail/prices?currencyCode=INR&\$filter=armRegionName%20eq%20%27centralindia%27%20and%20serviceName%20eq%20%27Azure%20Database%20for%20PostgreSQL%27"

Appendix B — raw staging Az inventory

Pulled via:

AZURE_CONFIG_DIR=~/.azure-ftl az resource list -g ftl-stg-rg-cin
AZURE_CONFIG_DIR=~/.azure-ftl az containerapp list -g ftl-stg-rg-cin -o json
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server list -g ftl-stg-rg-cin -o json
AZURE_CONFIG_DIR=~/.azure-ftl az redis list -g ftl-stg-rg-cin -o json
AZURE_CONFIG_DIR=~/.azure-ftl az consumption usage list --top 200

Appendix C — code citations

Every load-model claim in §4 carries a file:line against origin/release/staging. Re-verify with:

git -C ftl-backend show origin/release/staging:<path> | sed -n '<line>p'

Run inside /Users/amal/Projects/02-fifa26-trading-game/claude-code/repos/.

---

11. Corrections (appended 2026-06-11, same day)

After the operator pushed back on the LAW numbers, two facts changed the picture:

11.1 Original report assumed SDK sampling that was NOT in code

The §6 numbers for 10K and 50K user tiers carried a (sampled at 50 %) annotation on the Log Analytics line. That assumed SDK-side sampling was already wired. It was not. [Measured] internal/telemetry/telemetry.go on origin/release/staging at the time of the original draft used the OpenTelemetry SDK default — ParentBased(AlwaysSample()) = 100 %. No WithSampler(...) call, no SpanProcessor filter.

Honest pre-sampling numbers:

Tier	LAW line (corrected, NO sampling)	LAW line (originally quoted)	Delta
5K	~₹19 K /month	~₹19 K	0
10K	~₹185 K /month	~₹52 K	+₹133 K
50K	~₹500 K /month	~₹165 K	+₹335 K

The 5K-tier number was right because at that scale the assumed sampling rate was inside our daily quota anyway; the 10K and 50K errors are material.

11.2 Operator decision (2026-06-11)

Ship a custom ErrorPreservingProcessor at SDK level. 1 % ratio for non-error spans; 100 % retention for any span with Status.Code == codes.Error. No Azure-side ingestion sampling on top (that layer is uniform and would re-kill the pinned errors). Code change shipped via bug/otel-error-preserving-sampler → see paired ftl-backend PR.

This means: - Errors / exceptions kept at 100 % for incident debug. - Normal traces sampled to 1 % at the SDK before they ever leave the process. - Live debug paths (container log stream, AppInsights Live Metrics blade) are unsampled and free.

11.3 Corrected LAW lines per tier (with the new sampler shipped)

Tier	Total volume (raw)	After 1 % SDK sample + 100 % error retention	LAW cost / 39 days
5K	~3 GB/day	~0.3 GB/day, mostly errors	~₹35
10K	~8 GB/day	~0.8 GB/day	~₹400
50K	~25 GB/day	~3 GB/day	~₹3 300

These are now the realistic numbers. The §7 aggregate summary should be read against these LAW lines, not the originally-published ones.

11.4 New tier — 5K users / 3K concurrent, WC26 game-time scenario

Per operator request: 5 K total registered, 3 K concurrent during game-time, minimal compute outside game-time so users can browse profile / leaderboard / portfolio without burning idle replicas.

WC26 game-hour calc (measured against Sportmonks live this session):

Total fixtures pulled:    104
Days with fixtures:       35 of 39
Total match-hours:        161  (overlap-merged "any match live" hours)
Non-game hours:           775
Peak days (≥5 fixtures):  7
Days with 4 fixtures:     4
Days with 3 fixtures:     11
Avg match-hr / fixture-day: 4.6

Line-by-line for 5K/3K, 39-day window, with the new sampler:

Resource	Sizing	39-day INR
api-server game-time	2 × 0.5 vCPU/1 GiB × 161 hr × ₹5.22	1 681
api-server non-game	1 × 0.5 vCPU/1 GiB × 775 hr × ₹1.62	1 256
ws-server game-time	1 × 0.5 vCPU/1 GiB × 161 hr × ₹5.22	840
ws-server non-game	1 × 0.5 vCPU/1 GiB × 775 hr × ₹1.62	1 256
flusher game-time	1 × 0.5 vCPU/1 GiB × 161 hr × ₹5.22	840
flusher non-game	1 × 0.5 vCPU/1 GiB × 775 hr × ₹1.62	1 256
whatsapp-bot game-time	1 × 0.25 vCPU/0.5 GiB × 161 hr × ₹2.61	420
whatsapp-bot non-game	1 × 0.25 vCPU/0.5 GiB × 775 hr × ₹0.81	628
PG Flex compute	Standard_D2ds_v5 × 936 hr × ₹24.02	22 483
PG Flex storage	128 GiB × ₹12.54 × 39/30	2 087
PG Flex backup LRS	7-day, ~30 GiB × ₹9.09 × 39/30	355
Redis Standard C1	936 hr × ₹6.60	6 178
ACR Basic	39 days × ₹15.94	622
Log Analytics	1 % SDK sample, error-pinned	35
Static Web Apps	Standard pro-rated 39/30	1 119
Key Vault Standard	~65 K ops	52
ACS Email	5 K users × ~3 emails/mo × 39/30	390
Bandwidth egress	~150 GB × ₹10.53	1 579
TOTAL 39-day		~₹41 077

11.5 KEDA scale-to-zero analysis (NOT recommended for this tier)

The original §9 recommendation #3 mentioned scaling api + whatsapp-bot to zero during non-game hours. Saving is much smaller than first quoted (~₹1 500 not ₹4 700) because ws-server + flusher can't scale-to-zero (persistent WS connections + always-on dirty-set drainer respectively keep both warm). At ₹1.5 K saving over the tournament against a 3-8 s cold-start UX penalty on every browse-after-quiet-window, skip it unless a future review identifies a stronger trigger.

11.6 Live debugging paths are unsampled (clarification)

The 1 % sampler only affects retained Log Analytics history. Live debug paths are NOT sampled:

Path	Sampled?	Use case
az containerapp logs show --follow	No	Real-time stdout/stderr tail
AppInsights Live Metrics blade (portal)	No	Real-time telemetry stream during a session
az containerapp exec	No	Shell into the replica for ad-hoc inspection
KQL queries against Log Analytics history	Yes (1 %)	Post-incident forensic queries

For incidents in progress: open Live Metrics + container log stream. Both are free, unsampled, and reflect the current state of the system. Use sampled LAW queries for trend analysis and post-mortem dives.

11.7 Sub-confirmed inputs the operator declined

• 1-yr Reserved PG: not viable at the 39-day commitment horizon. Skipped per operator (2026-06-11).
• 3-yr Reserved PG: same — only relevant if prod runs past the tournament.
• Compounding Azure ingestion sampling on top of SDK 1 %: rejected — it's uniform and would kill the pinned errors. Operator approved single-layer SDK-only sampling.

---

12. Confidence summary (updated for §11)

• §2 staging inventory + MTD cost: [Measured] — pulled from az this session.
• §3 Azure unit prices: [Measured] — prices.azure.com API this session.
• §4 code-derived load model: [Measured] — git show origin/release/staging citations.
• §6 sizing logic per tier: [Estimated] from measured baselines.
• §11.3 corrected LAW lines: [Estimated] from [Measured] baseline ingestion rate × sampling math.
• §11.4 5K/3K WC26 tier: [Estimated] from [Measured] Sportmonks schedule + [Measured] Azure unit prices + [Estimated] sizing.
• §11.5 KEDA-skip recommendation: [Estimated] saving × [Estimated] UX cost.

---

13. Operator verification — what to check, when to check it, what's expected

This section gives the operator a runbook for verifying the bill matches the model as the tournament unfolds, plus the expected numbers at every checkpoint.

All numbers assume the 5K / 3K WC26 tier from §11.4 — 1 % SDK sampler shipped (PR #233), no KEDA scale-to-zero, no reserved PG. Adjust proportionally for your actual user count.

13.1 Cost dashboard / Cost Management query setup

Two ways to read MTD numbers:

A. Portal — Cost Management → Cost analysis
   Scope: subscription FootbalTradeLeage, filter ResourceGroup = ftl-prd-rg-cin
   Granularity: Daily
   View: Accumulated cost
   This is the easiest day-to-day check.

B. CLI — for scripted alerts or daily emails
   AZURE_CONFIG_DIR=~/.azure-ftl az consumption usage list \
     --start-date 2026-06-11 --end-date <today> \
     --query "[?resourceGroup=='ftl-prd-rg-cin'].{date:date, service:meterCategory, cost:cost}" \
     -o json | jq 'group_by(.service)
       | map({service: .[0].service, total: ([.[].cost | tonumber] | add)})'

C. Cost Management Budget alert (set once before launch)
   Threshold: ₹50 000 / month  (60 % of forecast for safety)
   Action: email + SMS to operator
   Path: portal → Cost Management → Budgets → Add

13.2 Per-day expected MTD checkpoints (5K / 3K, 39-day window)

After each milestone day, the cumulative MTD should sit within these bands:

End of day	Tournament day	Expected MTD	Acceptable band	Notes
2026-06-11	1	~₹1 050	₹950–₹1 200	Day 1 — single fixture
2026-06-13	3	~₹3 250	₹2 950–₹3 550	First weekend fully live
2026-06-18	8	~₹9 000	₹8 400–₹9 600	End of group-stage week 1
2026-06-25	15	~₹16 700	₹15 700–₹17 700	Mid-tournament checkpoint
2026-06-28	18	~₹20 000	₹18 900–₹21 100	Group stage complete
2026-07-04	24	~₹26 600	₹25 200–₹28 000	Round of 16 wrap
2026-07-12	32	~₹34 100	₹32 400–₹35 800	Quarter-finals complete
2026-07-19	39	~₹41 077	₹39 000–₹43 000	Final + 3rd place

A wider ±5 % band is fine — Azure billing has a 24-48 h settlement lag and bills don't post in real-time.

13.3 Per-week expected (assume Monday cutover)

Week #	Fixtures in window	Expected weekly ₹	Acceptable band
Pre-tournament	0	~₹6 100	₹5 700–₹6 500
W1 (Jun 11–17)	14	~₹8 100	₹7 600–₹8 600
W2 (Jun 18–24)	23	~₹8 200	₹7 700–₹8 700
W3 (Jun 25–Jul 1)	22 (group + R32 + R16)	~₹8 100	₹7 600–₹8 600
W4 (Jul 2–8)	12	~₹7 700	₹7 200–₹8 200
W5 (Jul 9–15)	5 (QF + SF)	~₹6 600	₹6 200–₹7 000
W6 (Jul 16–19)	2 (3rd-place + final)	~₹5 000	₹4 700–₹5 300

13.4 Per-line per-day verification

If MTD comes in outside the band above, drill into the specific line that drifted:

Line	/day target	KQL or az to verify the line
PG compute D2ds_v5	₹577	az postgres flexible-server show -g ftl-prd-rg-cin -n ftl-prd-pg-cin --query 'sku.name' → must be Standard_D2ds_v5
Redis Standard C1	₹158	az redis show -g ftl-prd-rg-cin -n ftl-prd-redis-cin --query '{sku:sku.name,cap:sku.capacity}' → must be {Standard,1}
Container Apps (4 apps)	₹210	az containerapp list -g ftl-prd-rg-cin --query '[].{n:name,min:properties.template.scale.minReplicas,max:properties.template.scale.maxReplicas,mode:properties.configuration.activeRevisionsMode}' → mode=Single on every row
PG storage	₹54	az postgres flexible-server show ... --query 'storage.storageSizeGb' → 128 (autoGrow may move this up)
PG backup	₹9	... --query 'backup.backupRetentionDays' → 7
Log Analytics (sampled)	₹0.90	KQL: Usage \| where TimeGenerated > ago(1d) \| summarize sum(Quantity) by Solution → ingestion GB/day. Should be ≪ 1 GB/day with sampler shipped.
Static Web Apps	₹29	az staticwebapp show -g ftl-prd-rg-cin --query 'sku.name' → Standard
ACR Basic	₹16	az acr show -g ftl-prd-rg-cin --query 'sku.name' → Basic
Key Vault	₹1.33	az keyvault show -g ftl-prd-rg-cin --query 'properties.sku.name' → standard
ACS Email	₹10	KQL: AzureCommunicationServices \| where TimeGenerated > ago(1d) \| count
Bandwidth	₹40	Cost Management → Cost analysis → filter MeterCategory = Bandwidth

13.5 Game-day vs dark-day variance — what to expect

Daily costs are not flat — game days bill more than dark days:

HEAVY GAME DAY (≥4 fixtures, ~8 match-hours)
  Always-on baseline        ₹844
  Container Apps active     ₹218
  LAW (matched events)      ₹2
  Bandwidth (live traffic)  ₹46
  ACS Email                 ₹10
  ─────────────────────────────
  TOTAL                     ~₹1 180 – ₹1 250

LIGHT GAME DAY (1–3 fixtures, ~3 match-hours)
  Always-on baseline        ₹844
  Container Apps active     ₹165
  LAW                       ₹1
  Bandwidth                 ₹40
  ACS Email                 ₹10
  ─────────────────────────────
  TOTAL                     ~₹1 050 – ₹1 150

DARK DAY (no fixtures)
  Always-on baseline        ₹844
  Container Apps idle       ₹136
  LAW                       ₹0.50
  Bandwidth                 ₹10
  ACS Email                 ₹5
  ─────────────────────────────
  TOTAL                     ~₹950 – ₹1 000

13.6 "Bill went over expected" — diagnosis flow

If MTD exceeds the band in §13.2 by more than 10 %, run this checklist top-to-bottom:

1. Container Apps revision drift  (May 2 audit incident)
   az containerapp show -g ftl-prd-rg-cin -n ftl-prd-api \
     --query 'properties.configuration.activeRevisionsMode' -o tsv
   Expected: Single
   If Multiple: rolling 9 revisions worth of warm replicas (cost = 9× normal)

2. PG SKU drift
   az postgres flexible-server show -g ftl-prd-rg-cin -n ftl-prd-pg-cin \
     --query 'sku.name' -o tsv
   Expected: Standard_D2ds_v5
   If D4ds_v5: somebody resized; check audit log

3. Redis tier drift
   az redis show -g ftl-prd-rg-cin -n ftl-prd-redis-cin \
     --query 'sku.{tier:name,cap:capacity}' -o json
   Expected: Standard C1
   If C2/C3: somebody resized

4. Log Analytics ingestion spike (sampler regressed / not deployed)
   KQL in Log Analytics workspace:
     Usage | where TimeGenerated > ago(7d)
           | summarize ingestedGB = sum(Quantity)/1024 by bin(TimeGenerated, 1d)
   Expected: <1 GB/day with sampler shipped
   If >5 GB/day: the sampler is not in the deployed image OR OTEL_TRACE_SAMPLE_RATIO is missing

5. NOISE_ENABLED was re-enabled by accident (code change broke comment-out)
   az containerapp show -g ftl-prd-rg-cin -n ftl-prd-api \
     --query "properties.template.containers[0].env[?name=='NOISE_ENABLED']" -o json
   Expected: value: "false"  (cosmetic — code is dead either way, but matches the docs)

6. SPORTMONKS_LEAGUE_IDS not set to WC26-only
   az containerapp show -g ftl-prd-rg-cin -n ftl-prd-api \
     --query "properties.template.containers[0].env[?name=='SPORTMONKS_LEAGUE_IDS']" -o json
   Expected: value: "732"  (= WC26 only; N_I drops 5,664 -> ~830 -> smaller Redis budget)

7. Bandwidth egress spike
   Cost Management → Cost analysis → MeterCategory = Bandwidth
   Expected: <1 GB/day during dark periods, <10 GB/day at peak
   If 100+ GB/day spike: CDN bypass or video / asset traffic; investigate via 
   AppGateway or front-door logs

8. Migrate job ran more than expected
   az containerapp job execution list -g ftl-prd-rg-cin -n ftl-prd-migrate \
     --query '[].{name:name, status:properties.status, started:properties.startTime}'
   Expected: 1 execution per release/staging or release/prod push
   If 10+: someone triggered manually in a loop

13.7 What it would cost to keep prod running one full calendar month (not just tournament)

Pre-tournament June 1–10 (10 days, no fixtures, all dark):

Always-on baseline      10 × ₹844     ₹8 440
Container Apps idle     10 × ₹136     ₹1 360
LAW (essentially zero)  10 × ₹0.50    ₹5
ACS Email                            ₹100
Bandwidth                            ₹100
                                     ───────
Pre-tournament addon (10 days)       ₹10 005

Post-tournament July 20–31 (12 days, all dark):

Same shape: 12 × ~₹1 000 = ₹12 000  (depending on idle-billing reality)

TOTAL FULL CALENDAR MONTH including tournament window (~5–6 weeks)
                      ₹41 077  (tournament)
                    + ₹10 005  (pre-fixture dark)
                    + ₹12 000  (post-final dark)
                    ──────────
                     ~₹63 100  for the full ~7-week window

Equivalent steady-state monthly burn if FTL kept running year-round at this sizing with no fixtures:

30 days × ₹950–1 000/day = ₹28 500 – ₹30 000 / month

(Drives the off-tournament economics: ~₹28 K/month idle floor with no users
playing. PG D2ds_v5 dominates. Scaling down to B2ms during off-season would
save ~₹17 K/month if you commit to a downgrade path.)

13.8 Different scenarios summary

For quick eyeball comparison after the launch metrics come in:

Scenario	39-day total	/day	/week
5K users / 3K conc — baseline (this report)	₹41 077	₹1 053	₹7 371
ditto + KEDA scale-to-zero non-game	₹39 577	₹1 015	₹7 102
ditto + Reserved 1-yr PG (only if >39 d)	₹33 200	₹851	₹5 956
10K users / 6K conc (rough extrapolation)	₹44 000	₹1 128	₹7 897
5K users + NO sampler shipped (regression)	₹45 000	₹1 154	₹8 077
5K users + PG drifted to D4ds_v5	₹52 000	₹1 333	₹9 333
5K users + Multi-revision drift on api×9	₹49 000	₹1 256	₹8 795

Use these as trip-wires when the dashboard MTD diverges from §13.2: the diff between the expected band and the actual usually points to a single drift cause from this table.

---

14. Current launch-readiness snapshot (2026-06-11, end-of-day)

Companion doc: prod-snapshot-2026-06-11.md — the measured prod inventory + MTD bill that pairs with the gap list below.

This section pins the actual end-state of the cost work as of 2026-06-11 EOD so an operator picking this up next can tell apart what has shipped, what is wired but inactive, and what is still outstanding before WC26 prod cutover.

14.1 What has shipped (live on staging + prod)

Item	State	Where
ErrorPreservingProcessor (custom OTel SpanProcessor)	Code merged + deployed to release/staging.	internal/telemetry/sampler.go (ftl-backend)
OTEL_TRACE_SAMPLE_RATIO=0.01 env var	Set on all three Container Apps on staging (ftl-stg-api, ftl-stg-ws, ftl-stg-flusher).	.github/workflows/deploy-staging.yml line ~210
Five unit tests covering the sampler	Pass in CI every deploy.	internal/telemetry/sampler_test.go
Prod LAW workspace daily cap	Set to 30 GB/day by operator, 2026-06-11 06:03 UTC.	ftl-prd-law-cin (dataIngestionStatus = RespectQuota)
Cost audit doc	Published at this URL with §1–§13.	this file

14.2 What is wired but inactive today

Item	What it would do	Why it's not firing today
ErrorPreservingProcessor	Drop 99 % of normal spans, keep 100 % of errors.	The OTel SDK only routes spans through the processor when an exporter is configured (OTEL_EXPORTER_OTLP_ENDPOINT set). That env var is unset on both staging and prod today (internal/telemetry/telemetry.go:78-91 installs a no-op TracerProvider when unset).
APPLICATIONINSIGHTS_CONNECTION_STRING	Authenticate the OTel→AppInsights direct exporter.	Connection string IS set (via Key Vault secretref) and captured by Config.AppInsightsConnStr, but the Microsoft Go AppInsights exporter is not yet GA — see ADR-0031. Today's intended path is via an OTel Collector sidecar.
Sampler ratio for spans authored by otelfiber.Middleware() and manual tracer.Start(...) call sites in the codebase.	Trace every HTTP request and inner span at 1 %.	Same reason: no-op TracerProvider means every tracer.Start(...) returns a free span; the processor never receives them.

Net effect today: the sampler is "pre-armed" in the binary, but nothing actually exports traces. The only LAW data flowing is Container Apps stdout (ContainerAppConsoleLogs_CL + ContainerAppSystemLogs_CL per the prod LAW Usage table). Prod LAW MTD as of this writing = ₹0 (under the 5 GB/month free-tier ceiling).

14.3 What is still outstanding before prod cutover

Task	Owner	Why it matters	If not done	Effort
Deploy OTel Collector sidecar + set OTEL_EXPORTER_OTLP_ENDPOINT on prod	ops + backend	Activates the trace export pipeline. Without it, the sampler we shipped is dead code and distributed traces in AppInsights stay empty.	All tracer.Start(...) spans become free function calls. AppInsights shows zero spans. Incident debugging falls back to container stdout + Live Metrics. The cost numbers in §6 / §11 / §13 STILL hold because no traces means no LAW ingestion cost. So skipping this is safe for the bill but bad for observability.	medium (1–2 days, includes ADR-0031 finalisation)
Set daily cap on staging LAW too	ops	Staging cap is already 0.9 GB/day from launch prep. Less urgent — already protected.	If a noisy bug ships and staging LAW spikes past 0.9 GB/day, ingestion drops until next UTC reset. Bill stays at ₹0.	done (cap was already there)
Set daily cap on prod LAW	ops	Prevents unbounded LAW bill in any failure mode.	If skipped: at 50K-user scale without sampler firing, raw container stdout alone could reach ~25 GB/day → ~₹165 K/month uncapped. With cap: capped at ₹6 603/day = ₹198 K/month max regardless.	DONE 2026-06-11 06:03 UTC (operator). dailyQuotaGb=30, RespectQuota active.
Set samplingPercentage on the AppInsights component to NULL (= no server-side sampling)	ops	The cost model assumes SDK-only sampling. Adding server-side sampling on top would re-uniformly-drop errors.	Today's value is already null on staging and prod (per Stream A inventory). No action unless someone changes it.	already correct — verify before cutover
Set SPORTMONKS_LEAGUE_IDS=732 on prod manifest	backend	Drops N_I from 5,664 → ~830 → frees Redis budget so Standard C1 fits with margin.	If skipped: N_I stays at ~5,664; decayPermanentImpact (every 60 s) does 5,664 Redis ops/min instead of 830. Bot engine multiplies it further. Memory: instrument hashes alone climb to ~7 MB (still fits C1) but dirty sets + idem keys get tighter. Functionally still works; cost of Redis upgrade if it stops fitting = ₹10 750/month for C2.	one env-var in deploy-prod.yml
Verify Single-revision mode active on every prod Container App	ops	Prevents the May 2 audit drift incident (9 warm revisions × normal cost).	If skipped: a future az containerapp update without --revision-suffix flag risks flipping mode. Real-world cost: 9× burn (~₹40 K extra/month on ftl-prd-api alone).	az containerapp show --query 'properties.configuration.activeRevisionsMode' on each app; expect Single.
Decide on KEDA scale-to-zero for api + whatsapp-bot during non-game hours	ops + product	Saves ~₹1 500 over the 39-day tournament.	If skipped: pay the ₹1 500. Acceptable given the cold-start UX cost (3–8 s on first request after quiet window). Recommended skip per §11.5.	n/a (decision: skip)
Reserved PG (1-yr / 3-yr) commitment	finance	35 % / 60 % off PG compute.	If skipped during the 39-day tournament alone: pay ₹22 483 vs ₹14 600 (1-yr) or ₹9 000 (3-yr). At 39-day commitment horizon the break-even doesn't apply — only relevant if prod runs past the tournament.	Recommended SKIP per operator (2026-06-11). Revisit post-tournament if prod stays up.

14.4 Decision-tree: what if OTel Collector sidecar is NOT shipped before WC26?

This is the most consequential outstanding item. Below is the honest cost-and-debug picture for each path:

PATH A — Skip OTel Collector. Run prod with OTel as no-op.
─────────────────────────────────────────────────────────
Cost impact:
  - LAW cost stays near zero (only container stdout → LAW).
  - Tournament-total prod LAW ≈ ₹0–500 over 39 days.
  - 5K/3K tournament TOTAL = ~₹41 077 (matches §11.4).
  - Even at 50K users, LAW stays bounded by container stdout volume
    + the 30 GB/day cap that's already in place.

Observability impact:
  - AppInsights distributed traces: EMPTY.
  - No span timing. No KQL on request flow.
  - Incident debug = container stdout (sliced via KQL on
    ContainerAppConsoleLogs_CL) + Live Metrics + az containerapp logs.
  - Live Metrics shows real-time metrics + stack traces at the moment
    they happen, but cannot be replayed historically.

Suitability:
  - Acceptable for launch IF the team is comfortable debugging via
    structured slog output rather than spans.
  - Most FTL services emit reasonable slog at INFO+ (events, lua results,
    errors) — slog should cover ~80 % of incident questions.

PATH B — Ship OTel Collector sidecar before cutover.
─────────────────────────────────────────────────────────
Cost impact:
  - SDK sampler activates: 99 % of normal spans dropped, 100 % of errors kept.
  - 5K/3K tournament LAW from spans: ~₹35 over 39 days (matches §11.3).
  - Tournament TOTAL still = ~₹41 077.

Observability impact:
  - Full distributed traces in AppInsights.
  - End-to-end transaction view (api → PG, ws → Redis, Lua call timings).
  - Errors at 100 % retention → every incident has a captured trace.

Suitability:
  - Strongly preferred for any production system at >1K users.
  - 1–2 days of work (ADR-0031 spec + Collector container + manifest).
  - Cap of 30 GB/day still in place as insurance — if sampler regresses,
    bill is bounded.

RECOMMENDATION:
  Ship Path B in the 2026-06-11 → 2026-06-12 window before WC26 traffic
  starts ramping. If you cannot fit the work in, Path A is operationally
  safe — you'll have a working game, just thinner observability.

14.5 Verification commands the operator should run on prod before launch

# 1. Confirm prod LAW cap is in place
AZURE_CONFIG_DIR=~/.azure-ftl az monitor log-analytics workspace show \
  -g ftl-prd-rg-cin -n ftl-prd-law-cin \
  --query 'workspaceCapping' -o json
# Expected: dailyQuotaGb: 30, dataIngestionStatus: RespectQuota

# 2. Confirm prod Container Apps in Single-revision mode
AZURE_CONFIG_DIR=~/.azure-ftl az containerapp list -g ftl-prd-rg-cin \
  --query '[].{name:name, mode:properties.configuration.activeRevisionsMode}' -o tsv
# Expected: every row ends with 'Single'

# 3. Confirm prod PG SKU + autoGrow
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server show \
  -g ftl-prd-rg-cin -n ftl-prd-pg-cin \
  --query '{sku:sku.name, autoGrow:storage.autoGrow, storage:storage.storageSizeGb}' -o json
# Expected: D2ds_v5 (or D4 if pre-upgraded), autoGrow: Enabled, storage: ≥128 GiB

# 4. Confirm OTEL_TRACE_SAMPLE_RATIO is set on all three apps after prod deploy
for app in ftl-prd-api ftl-prd-ws ftl-prd-flusher; do
  AZURE_CONFIG_DIR=~/.azure-ftl az containerapp show -g ftl-prd-rg-cin -n "$app" \
    --query "properties.template.containers[0].env[?name=='OTEL_TRACE_SAMPLE_RATIO']" -o json
done
# Expected: [{"name":"OTEL_TRACE_SAMPLE_RATIO","value":"0.01"}] on each.

# 5. Confirm SPORTMONKS_LEAGUE_IDS is WC26-only
AZURE_CONFIG_DIR=~/.azure-ftl az containerapp show -g ftl-prd-rg-cin -n ftl-prd-api \
  --query "properties.template.containers[0].env[?name=='SPORTMONKS_LEAGUE_IDS']" -o json
# Expected: [{"name":"SPORTMONKS_LEAGUE_IDS","value":"732"}]

14.6 Honest summary

What you can safely launch with TODAY:
  ✓ Container Apps healthy on staging (3 replicas restartCount=0)
  ✓ ErrorPreservingProcessor shipped + tested + deployed to release/staging
  ✓ Prod LAW cap = 30 GB/day (≈₹6 603/day worst case)
  ✓ Cost model for WC26 = ~₹41 077 for 5K/3K tier over 39 days

What launch can still surprise you on:
  ⚠ OTel traces are no-op until OTLP endpoint is wired (Path A vs B above).
  ⚠ Prod PG is still B2ms — needs the SKU upgrade per §1 / §8 of the
    earlier WC26 deploy notes.
  ⚠ Prod manifest may not have SPORTMONKS_LEAGUE_IDS=732 set yet.
  ⚠ Prod Single-revision mode should be re-verified post any az containerapp
    update — drift caused the May 2 incident.

What is guaranteed safe:
  ✓ No unbounded LAW bill (cap is active and confirmed by `RespectQuota`).
  ✓ No code-path regressions (full Go test suite green on the deploy commit).
  ✓ No breaking change to event ingestion (no-op telemetry means
    behaviour identical to pre-sampler image).