title: Incident Debugging (current config, no OTel) status: active last-updated: 2026-06-11 authors: [@amalkrsihna] audience: [backend, ops, on-call] abstract: > How to triage prod incidents using the live tools the system actually has today — container log stream, Live Metrics, Cost Mgmt, raw az, KQL on Log Analytics, SQL on PG, redis-cli. No OTel traces. Per-symptom triage flowcharts. companion: [prod-snapshot-2026-06-11, usage-analytics]

Incident Debugging (current config — no OTel)¶

Companion: prod-snapshot-2026-06-11, usage-analytics, prod-launch-runbook.

Operator decision 2026-06-11: OTel Collector sidecar is NOT shipped for WC26 launch. Distributed traces in AppInsights stay empty. Debugging uses the tools below. The 1 % SDK sampler shipped is pre-armed for whenever OTLP is wired later — not used now.

0. Mental model¶

Without OTel traces, you have four signal sources for prod debugging:

┌─────────────────────────────────────────────────────────────────┐
│ 1. CONTAINER STDOUT/STDERR                                       │
│    Source: Go slog at INFO+ level                                │
│    Routes to: Container Apps log stream + Log Analytics          │
│    Best for: errors, panics, scheduled-task heartbeats           │
│    Latency: real-time (stream) / ~2-3 min (LAW ingestion lag)    │
│                                                                  │
│ 2. AZURE LIVE METRICS (App Insights blade)                        │
│    Source: AppInsights connection string + Live Stream feature    │
│    Routes to: portal UI only                                     │
│    Best for: real-time request rate, exceptions, dependencies     │
│    Latency: < 1 s                                                 │
│    Today: NOT WIRED (connection string not on prod env)           │
│                                                                  │
│ 3. AZURE METRICS (CPU/memory/disk/IOPS)                           │
│    Source: platform-level resource telemetry                      │
│    Routes to: Metrics blade per resource                          │
│    Best for: SKU saturation, autoGrow events, replica counts      │
│    Latency: ~1 min                                                │
│                                                                  │
│ 4. DATA STORE INSPECTION                                          │
│    PG (via psql, pgAdmin, az exec)                                │
│    Redis (via az redis-cli or local script with redis-cli)        │
│    Best for: data inconsistency, queue depth, position state       │
│    Latency: immediate                                              │
└─────────────────────────────────────────────────────────────────┘

1. First five minutes of any incident¶

Before deep-diving, gather these in order:

# 1. Are services up?
for app in ftl-prd-api ftl-prd-ws ftl-prd-flusher; do
  echo "=== $app ==="
  AZURE_CONFIG_DIR=~/.azure-ftl az containerapp replica list \
    -g ftl-prd-rg-cin -n "$app" \
    --query '[].{name:name, state:properties.runningState, restarts:properties.containers[0].restartCount}' -o tsv
done
# Expected: all Running, 0 restarts. Restarts > 0 → crashloop.

# 2. Are HTTP probes green?
curl -sw "\nHTTP %{http_code} | %{time_total}s\n" https://prd-api.ftljeta.cloud/health
curl -sw "\nHTTP %{http_code}\n" https://prd-ws.ftljeta.cloud/health

# 3. Recent errors in stdout?
AZURE_CONFIG_DIR=~/.azure-ftl az containerapp logs show \
  -n ftl-prd-api -g ftl-prd-rg-cin --tail 100 --format json | \
  grep -iE 'panic|fatal|level=error|FATAL' | head -20

# 4. PG and Redis up?
AZURE_CONFIG_DIR=~/.azure-ftl az postgres flexible-server show \
  -g ftl-prd-rg-cin -n ftl-prd-pg-cin --query '{state:state, sku:sku.name}' -o tsv
AZURE_CONFIG_DIR=~/.azure-ftl az redis show \
  -g ftl-prd-rg-cin -n ftl-prd-redis-cin --query '{state:provisioningState, sku:sku.name}' -o tsv

# 5. Recent revision rolls?
for app in ftl-prd-api ftl-prd-ws ftl-prd-flusher; do
  AZURE_CONFIG_DIR=~/.azure-ftl az containerapp revision list \
    -g ftl-prd-rg-cin -n "$app" \
    --query 'sort_by(@, &properties.createdTime)[-3:].{name:name, active:properties.active, created:properties.createdTime}' -o table
done

2. Live debugging tools¶

2.1 Container log stream (always works)¶

CLIPortal

# Tail the latest 300 lines on api server
AZURE_CONFIG_DIR=~/.azure-ftl az containerapp logs show \
  -n ftl-prd-api -g ftl-prd-rg-cin --tail 300 --format json

# Real-time follow (5-min window)
AZURE_CONFIG_DIR=~/.azure-ftl az containerapp logs show \
  -n ftl-prd-api -g ftl-prd-rg-cin --follow --tail 0

Portal → ftl-prd-api → Monitoring → Log stream.
Select the running revision and replica.
Click Connect. Live stream appears.
Use the search bar in the top-right to filter.

Cost: zero. Stream bypasses Log Analytics ingestion entirely.

Limit: only shows recent lines; historical queries need KQL on Log Analytics workspace (next section).

2.2 Live Metrics blade (NOT WIRED — requires fix)¶

Today the APPLICATIONINSIGHTS_CONNECTION_STRING env var is unset on all three prod Container Apps. Live Metrics will be empty until prod-launch-runbook.md §3.3 is run.

Once wired:

Portal → ftl-prd-ai-cin → Investigate → Live Metrics.
Connects to the running replicas. Shows real-time:
Incoming request rate + p50/p95/p99 latency
Dependency call rate (PG, Redis)
Exceptions / failed requests
CPU + memory per server
Sample down to specific request URLs / response codes.

This is the highest-fidelity real-time tool. Use it FIRST during an incident once the wiring is fixed.

2.3 Metrics blade (resource-level)¶

Per-resource CPU/memory/disk/IOPS. No code change needed; always works.

PortalCLI

For PG: 1. Portal → ftl-prd-pg-cin → Monitoring → Metrics. 2. Add metric: CPU percent, Memory percent, Storage percent, Active Connections, Read IOPS, Write IOPS. 3. Window: last 1 hour.

For Redis: 1. Portal → ftl-prd-redis-cin → Monitoring → Metrics. 2. Add: Server Load (%), Used Memory (%), Cache Hits, Cache Misses, Connected Clients.

For Container Apps: 1. Portal → ftl-prd-api → Monitoring → Metrics. 2. Add: CPU Usage, Memory Working Set, Replica Count, Restart Count.

# Last 1 h PG CPU
AZURE_CONFIG_DIR=~/.azure-ftl az monitor metrics list \
  --resource /subscriptions/8c48ba4e-7384-4579-a6e2-9aeb6c5a5de9/resourceGroups/ftl-prd-rg-cin/providers/Microsoft.DBforPostgreSQL/flexibleServers/ftl-prd-pg-cin \
  --metric "cpu_percent" --interval PT1M --start-time "$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)" \
  --query 'value[0].timeseries[0].data[].{t:timeStamp,v:average}' -o table

# Same for Redis
AZURE_CONFIG_DIR=~/.azure-ftl az monitor metrics list \
  --resource /subscriptions/8c48ba4e-.../resourceGroups/ftl-prd-rg-cin/providers/Microsoft.Cache/Redis/ftl-prd-redis-cin \
  --metric "percentProcessorTime" --interval PT1M --start-time "..." -o table

3. KQL queries against Log Analytics¶

The workspace is ftl-prd-law-cin (customerId 70e5df86-b238-4a05-a4a3-503ffae12533). Open Portal → workspace → Logs to run these interactively.

3.1 Recent errors¶

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(1h)
| where Log_s contains 'level=error' or Log_s contains 'panic' or Log_s contains 'FATAL'
| project TimeGenerated, ContainerAppName_s, Log_s
| order by TimeGenerated desc
| take 100

3.2 Error rate over time¶

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(6h)
| extend isErr = iff(Log_s contains 'level=error' or Log_s contains 'panic', 1, 0)
| summarize total = count(), errors = sum(isErr) by bin(TimeGenerated, 5m), ContainerAppName_s
| extend errorPct = todouble(errors) / total * 100
| order by TimeGenerated desc

3.3 Sportmonks poller activity¶

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(30m)
| where ContainerAppName_s == 'ftl-prd-api'
| where Log_s contains 'sportmonks' or Log_s contains 'PollLivescores' or Log_s contains 'rate_limit'
| project TimeGenerated, Log_s
| order by TimeGenerated desc
| take 50

3.4 Trade execution¶

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(15m)
| where ContainerAppName_s == 'ftl-prd-api'
| where Log_s contains 'position_open' or Log_s contains 'position_close' or Log_s contains 'cooldown_active' or Log_s contains 'PRICE_MOVED'
| project TimeGenerated, Log_s
| order by TimeGenerated desc

3.5 WebSocket churn¶

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(30m)
| where ContainerAppName_s == 'ftl-prd-ws'
| where Log_s contains 'conn' or Log_s contains 'subscribe' or Log_s contains 'unsubscribe' or Log_s contains 'write_timeout'
| summarize count() by bin(TimeGenerated, 1m)
| order by TimeGenerated desc

3.6 Container restarts¶

ContainerAppSystemLogs_CL
| where TimeGenerated > ago(24h)
| where EventName_s contains 'Restart' or Reason_s contains 'CrashLoopBackOff'
| project TimeGenerated, ContainerAppName_s, EventName_s, Reason_s, Log_s
| order by TimeGenerated desc

3.7 LAW daily cost burn (sanity check)¶

Usage
| where TimeGenerated > ago(7d)
| summarize ingestedGB = sum(Quantity)/1024.0 by bin(TimeGenerated, 1d), DataType
| order by TimeGenerated desc, ingestedGB desc

If a single day shows > 5 GB, investigate — usually a noisy bug or a regression. Below ~1 GB/day under normal load with the sampler.

4. SQL queries on Postgres¶

Connect via psql $PROD_DATABASE_URL (DATABASE_URL secret in KV) or run from a temporary IP-whitelisted client. Read-only by default; never run UPDATE/DELETE on prod without operator + this runbook's approval.

4.1 Recent trade activity¶

-- Last 100 trades + their side
SELECT id, user_id, instrument_id, direction, lot_size, open_price, opened_at_ms, closed_at_ms,
       realized_pnl, close_reason
FROM cfd_positions
ORDER BY opened_at_ms DESC
LIMIT 100;

4.2 Trade volume per hour¶

SELECT date_trunc('hour', to_timestamp(opened_at_ms/1000)) AS hour,
       COUNT(*) AS opened,
       COUNT(CASE WHEN closed_at_ms IS NOT NULL THEN 1 END) AS closed,
       SUM(lot_size) AS total_lots
FROM cfd_positions
WHERE opened_at_ms > extract(epoch from NOW() - INTERVAL '24 hours') * 1000
GROUP BY 1
ORDER BY 1 DESC;

4.3 Currently open positions¶

SELECT direction, COUNT(*), SUM(lot_size)
FROM cfd_positions
WHERE closed_at_ms IS NULL
GROUP BY direction;

4.4 Stuck positions (open but instrument has no recent price tick)¶

SELECT p.id, p.user_id, p.instrument_id, p.opened_at_ms,
       (SELECT MAX(ts) FROM price_ticks WHERE instrument_id = p.instrument_id) AS last_tick
FROM cfd_positions p
WHERE p.closed_at_ms IS NULL
  AND (SELECT MAX(ts) FROM price_ticks WHERE instrument_id = p.instrument_id)
      < NOW() - INTERVAL '5 minutes'
LIMIT 50;

4.5 Wallet sanity check¶

-- Wallets with negative balance (should never happen)
SELECT user_id, balance, updated_at
FROM wallets
WHERE balance < 0
ORDER BY balance ASC;

4.6 Migration state¶

SELECT version, dirty FROM schema_migrations ORDER BY version DESC LIMIT 5;

dirty = TRUE on the latest row → a migration failed mid-flight. Resolve before any further deploy (prod-deployment-runbook.md §7).

4.7 Recent events / match feed¶

SELECT fixture_id, provider_event_id, event_type, minute, player_id, created_at
FROM match_events
ORDER BY created_at DESC
LIMIT 100;

4.8 Sportmonks event idempotency check¶

-- Should always return 0; if > 0 then event_bump_log unique constraint is broken
SELECT fixture_id, provider_event_id, instrument_id, COUNT(*)
FROM event_bump_log
GROUP BY fixture_id, provider_event_id, instrument_id
HAVING COUNT(*) > 1;

4.9 Slow queries (active right now)¶

SELECT pid, now() - pg_stat_activity.query_start AS duration, state, query
FROM pg_stat_activity
WHERE state != 'idle' AND query NOT LIKE '%pg_stat_activity%'
ORDER BY duration DESC
LIMIT 20;

Kill a runaway: SELECT pg_cancel_backend(<pid>); (graceful) or SELECT pg_terminate_backend(<pid>); (force). Operator approval required.

4.10 Connection pool saturation¶

SELECT COUNT(*), state FROM pg_stat_activity GROUP BY state;

Expected active + idle well below the SKU's max-connections (D2ds_v5 default = 1671 conns).

5. Redis inspection¶

Connect from a whitelisted IP. The host is ftl-prd-redis-cin.redis.cache.windows.net:6380 (TLS) with the primary key from ftl-prd-kv-cin/redis-url.

5.1 Keyspace overview¶

redis-cli -h ftl-prd-redis-cin.redis.cache.windows.net -p 6380 --tls -a "$REDIS_KEY" INFO keyspace
redis-cli -h ftl-prd-redis-cin.redis.cache.windows.net -p 6380 --tls -a "$REDIS_KEY" DBSIZE

5.2 Active WS subscribers per instrument (sample)¶

# This data isn't in Redis — it's in-process per ws-server replica. Use the ws /health endpoint
curl -s https://prd-ws.ftljeta.cloud/health | jq
# Returns: { "connections": N, "subscriptions": M }
# Staging: https://ws.ftljeta.cloud/health (no `prd-` prefix)

5.3 Dirty-set depth (flusher backlog)¶

for key in dirty:instruments dirty:wallets dirty:positions; do
  echo "$key: $(redis-cli -h ... SCARD $key)"
done

Expected: < 1000 each. > 10000 means flusher fell behind — check flusher logs for PG connection errors.

5.4 Idempotency cache size¶

redis-cli -h ... --scan --pattern 'idem:*' | wc -l
redis-cli -h ... --scan --pattern 'idem_result:*' | wc -l

5.5 Sportmonks dedup sets¶

redis-cli -h ... --scan --pattern 'seen_events:*' | head -20
redis-cli -h ... --scan --pattern 'seen_timeline:*' | head -20

Each fixture has its own set; check SCARD on an active match-day fixture to see how many events ingested.

6. Per-symptom playbook¶

6.1 "User can't log in"¶

Container log stream on api: filter for auth / google / oauth.
KQL §3.1 for any panic on api.
Check Google OAuth client ID secret rotation status (KV: ftl-prd-kv-cin/google-client-secret).
PG §4.5: check wallets table for the user's row.
Live Metrics (when wired): failed requests on /api/auth/* routes.

6.2 "Trade button is greyed out / 5xx on open"¶

Container log stream: position_open / PRICE_MOVED / instrument_frozen.
SQL §4.3: confirm user has open-position headroom (max_open_positions per CLAUDE.md is high; usually not the cause).
Redis: redis-cli HGET instrument:<uuid> frozen — if 1, instrument is frozen; check why.
Metrics blade on api: CPU / memory near 100 %?
PG §4.10: connection pool saturated?

6.3 "Close button says 'cooldown active' on every position"¶

Confirm the per-position 180 s cooldown is intended behaviour for recent opens (see CFD spec).
Container log stream: search cooldown_active.
SQL: SELECT id, opened_at_ms FROM cfd_positions WHERE user_id = '<X>' AND closed_at_ms IS NULL; then compare against extract(epoch FROM NOW()) * 1000 — should be > 180 000 ms apart for closes to succeed.
Lua script unchanged confirmation: redis-cli SCRIPT EXISTS <sha-of-position_close.lua>.

6.4 "Price chart frozen"¶

WS /health: connections count drop?
Container log stream on ws: filter for write_timeout, subscriber goroutine.
Sportmonks poller: KQL §3.3 — is it polling? If rate_limit.remaining=0, key exhausted.
PG §4.7: are new match_events landing? If not, ticker is wedged.
Redis: redis-cli MGET instrument:<uuid>:live_base_price instrument:<uuid>:micro_bump to confirm the back-end values are moving.

6.5 "Bill spike alert"¶

Cost Management → Cost analysis → group by service → identify the line.
KQL §3.7: LAW daily burn — is it sampler regression or container stdout spike?
Metrics blade on PG / Redis: CPU/memory near 100 % (under-sized)?
Container Apps revision count: drift to Multiple-revision mode (May 2 audit pattern)?

6.6 "Match events not showing up in stadium feed"¶

Sportmonks direct test:

SM_KEY=$(AZURE_CONFIG_DIR=~/.azure-ftl az keyvault secret show \
  --vault-name ftl-prd-kv-cin --name sportmonks-api-key --query value -o tsv)
curl -s "https://api.sportmonks.com/v3/football/livescores/latest?api_token=${SM_KEY}&include=events.type;timeline" | jq '.rate_limit'

Confirm remaining > 100.

Container log stream on api: filter processEvents / processTimeline.
Redis: redis-cli LLEN match_events:<fixture_id> — populated?
SQL §4.7: PG durable copy populated?

6.7 "Container Apps replica restartCount > 0"¶

KQL §3.6 for the most recent crash event.
Container logs preceding the restart (look at the last 50 lines of the killed revision).
Metrics blade: memory hit the limit? OOM kill leaves no go panic trace.
Check livenessProbe / readinessProbe configuration — health endpoint may be returning > 200 ms which trips the probe.

7. Known limitations of this debug toolkit¶

Limitation	Workaround
No distributed traces	Use slog request IDs (look for `request_id=<uuid>` in stdout).
Live Metrics not wired (yet)	Use container log stream + Azure Metrics blade.
No frontend telemetry (today)	Browser DevTools Network tab; in-page error toasts.
LAW has 2-3 min ingestion lag	For real-time, use container log stream.
Redis Basic C0 (pre-upgrade) cap 250 MB	After `prod-launch-runbook.md` §3.5 → C1 1 GB.
PG B2ms credit cliff (pre-upgrade)	After §3.4 → D2ds_v5 dedicated.

8. Escalation path¶

Tier 1 — Self-serve (this runbook + KQL)
  ↓ if symptom persists > 15 min after first attempt
Tier 2 — Open ops Slack channel, page on-call backend dev
  ↓ if data-loss / financial impact suspected
Tier 3 — Pause new trades via admin endpoint, file Azure support ticket
  ↓ if Azure platform issue confirmed
Tier 4 — Public status page update, prepare PIR