Skip to content

adr: 0029 title: "WS broadcast worker pool — Option C, W=16, heartbeat 30s/45s, drop-on-backpressure" status: accepted implementation-status: pending date: 2026-06-08 implemented-date: null implemented-in: - ftl-backend feat/ws-worker-pool - ftl-frontend feat/ws-worker-pool deciders: [@amalkrsihna] affects-specs: [realtime] affects-code: - ftl-backend/internal/ws/hub.go - ftl-backend/cmd/ws-server/main.go - ftl-frontend/src/components/SelectivePriceSubscriber.tsx - ftl-frontend/src/stores/positions.ts - ftl-frontend/src/stores/stadium.ts - ftl-frontend/src/stores/ui.ts supersedes: null superseded-by: null

ADR-0029: WS broadcast worker pool — Option C, W=16

Context

Measured bottleneck

The WS subscriber goroutine in subscriber.go:Start() is a single goroutine for-select loop. For each Redis price:* message it calls hub.Broadcast(update), which marshals the update once and then iterates synchronously over all connections subscribed to that instrument:

for conn → writeWithDeadline(conn, BinaryMessage, data, 100ms)

At 5,000 connected clients subscribing to all 5,664 instruments (the current GlobalPriceSubscriber behavior), and 2 simultaneous matches producing price updates at the 10-second poll interval:

  • Price frames per second: 5,664 instruments / 10s = ~566 frames/s
  • Writes per frame (5K clients): 5,000 writes
  • Total writes/second: 566 × 5,000 = 2,830,000 writes/s

Wait — this is the theoretical maximum. In practice with selective instrument subscriptions from active clients: measured at 52,000 writes/s for 2 simultaneous matches (verified: reports/wc26-final-lock-2026-06-08/03c-ws-db-admin-audit.md §1.3).

Single goroutine throughput at 0.3ms/write: 3,333 writes/s. Overload factor at 5K subscribe-all: 15.6×.

At 4 simultaneous Group Stage matches (possible on WC26 day 1), the per-Broadcast call conns list grows and the inter-call window shrinks to 48ms. W=8 produces a 62.5ms drain time — exceeding the window. W=16 produces 31.3ms — safely within it.

Server-side hub filter already works

hub.subscriptions[idx] contains only connections that sent {"action":"subscribe","instrumentId":"<uuid>"} for that specific instrument. The current behavior saturates the subscriber goroutine only because the frontend (GlobalPriceSubscriber.tsx:44) subscribes all 5,664 instruments indiscriminately.

Decision

Phase 1 — Broadcast-per-frame parallel dispatch (Option C), W=16

Three sharding options were evaluated:

Option A: Shard by instrument idx. Worker W_i handles all connections for instruments where idx % W == i. Does not help: the subscriber goroutine still calls Broadcast sequentially and the goroutine blocks waiting for each Broadcast call to drain before reading the next Redis message.

Option B: Shard by connection (round-robin). Each worker owns a fixed subset of connections across all instruments. Same fundamental problem: the subscriber goroutine calls Broadcast sequentially per Redis message; the goroutine blocks while the worker drains that worker's connection slice. Workers running in parallel on their own slices do not help the subscriber goroutine return to the Redis channel faster.

Option C: Broadcast-per-frame parallel dispatch. Each hub.Broadcast call fans out to W goroutines via buffered channels simultaneously, then returns immediately to the caller. The W goroutines drain in parallel; the subscriber goroutine is unblocked after dispatching and can consume the next Redis message. This is the only design that parallelizes the actual write work within one Broadcast call.

Chosen: Option C.

Implementation (hub.go)

type BroadcastJob struct {
    data  []byte
    conns []*websocket.Conn
}

type Hub struct {
    // ... existing fields ...
    broadcastWorkers []chan BroadcastJob
    workerCount      int
}

func (h *Hub) Broadcast(update *PriceUpdate) {
    data, err := msgpack.Marshal(update)
    if err != nil { return }

    h.mu.RLock()
    origSubs := h.subscriptions[update.InstrumentIdx]
    conns := make([]*websocket.Conn, 0, len(origSubs))
    for conn := range origSubs { conns = append(conns, conn) }
    h.mu.RUnlock()

    if len(conns) == 0 { return }
    telemetry.WSBroadcastTotal.Add(float64(len(conns)))

    W := h.workerCount
    slices := make([][]*websocket.Conn, W)
    for i, conn := range conns { slices[i%W] = append(slices[i%W], conn) }
    for i, slice := range slices {
        if len(slice) == 0 { continue }
        select {
        case h.broadcastWorkers[i] <- BroadcastJob{data: data, conns: slice}:
        default:
            telemetry.WSBroadcastDrops.Add(float64(len(slice)))
        }
    }
}

Each worker goroutine (lifetime = process, started via StartWorkers(ctx)) reads from its own channel and writes to each connection. Write failure triggers eviction: UnsubscribeAll + RemoveConn.

W=16 — rationale: W=16 covers 5K subscribe-all at the 48ms inter-call window (4 simultaneous matches) with drain time 31.3ms. W=8 covers 5K at 2 matches (drain 62.5ms vs 48ms inter-call — marginal). W=16 provides headroom for the 4-match Group Stage peak without requiring an emergency config change between matches. The value is read from env var WS_WORKER_COUNT (default 16); can be raised to 32 post-launch without a deploy.

Channel buffer: 64 jobs/worker. At W=16 and 5K subscribe-all with 2 simultaneous matches, the dispatch rate per worker is approximately 1.3 jobs/second (10.4 Broadcast calls/s total, 16 workers). The buffer of 64 provides 49 seconds of headroom before backpressure — far exceeding any realistic burst.

Drop policy (backpressure)

If a worker channel is full at dispatch time, the Broadcast call for that partition is dropped and ws_broadcast_drops_total{worker=i} is incremented. This is preferred over blocking: blocking in Broadcast would propagate backpressure to the subscriber goroutine, which would then fall behind on the Redis channel and miss price frames entirely. Dropping individual writes to a slow client's partition is the lesser evil — the client misses one price tick and receives the next one 10 seconds later.

Alert threshold: ws_broadcast_drops_total rate > 100/min triggers a PagerDuty alert (pool saturation warning, investigate connection latency).

Heartbeat — 30s ping / 45s read deadline

Dead reader connections (clients that have closed the browser without sending a WS close frame) are currently discovered only on the next write failure (100ms write deadline). With the pool, write failure still triggers eviction. However, connections that are subscribed to no instruments (or only instruments that never tick) can persist indefinitely without being discovered.

Add a 30-second ping ticker on the read-loop goroutine:

conn.SetReadDeadline(time.Now().Add(45 * time.Second))
conn.SetPongHandler(func(string) error {
    conn.SetReadDeadline(time.Now().Add(45 * time.Second))
    return nil
})
go func() {
    ticker := time.NewTicker(30 * time.Second)
    defer ticker.Stop()
    for range ticker.C {
        if err := conn.WriteControl(websocket.PingMessage, nil,
            time.Now().Add(10*time.Second)); err != nil {
            return
        }
    }
}()

45-second read deadline: if no pong is received within 15 seconds of a ping (ping every 30s, deadline 45s), the connection is evicted. This bounds the maximum number of ghost connections to those that have been dead for < 45 seconds at any point.

Metrics

Metric Type Labels Purpose
ws_broadcast_total Counter — Total write attempts dispatched
ws_broadcast_drops_total Counter worker Writes dropped — full channel
ws_worker_queue_depth Gauge worker Jobs buffered per worker channel
ws_connections_total Gauge — physConns (already exists)
ws_evictions_total Counter — Connections evicted on write failure

ws_worker_queue_depth is polled every 5 seconds via a background goroutine. At normal load, all depths should be 0 or near-0; sustained > 32 → warning alert.

Phase 2 — Selective subscription (Week-1, post-launch)

GlobalPriceSubscriber.tsx subscribes all 5,664 instruments, producing 52,000 writes/s at 5K subscribe-all. Replacing it with SelectivePriceSubscriber.tsx limits each client to ~80 instruments (44 Stadium lineup + 10 open positions + watchlist), reducing total writes/s to ~730 — a 71× reduction. At that rate, the single subscriber goroutine is sufficient and the worker pool becomes pure headroom.

SelectivePriceSubscriber.tsx derives the subscription set from three stores:

const subscriptionSet = useMemo(() => {
    const ids = new Set<string>();
    openPositionInstrumentIds.forEach(id => ids.add(id));
    stadiumInstrumentIds.forEach(id => ids.add(id));
    if (viewedInstrumentId) ids.add(viewedInstrumentId);
    return Array.from(ids);
}, [openPositionInstrumentIds, stadiumInstrumentIds, viewedInstrumentId]);

Required store additions (minimal):

Store New field Populated by
stores/positions.ts openInstrumentIds: string[] Derived from positions array on open/close
stores/stadium.ts visibleInstrumentIds: string[] Set by Stadium component on mount/fixture-change
stores/ui.ts instrumentId: string \| null Set by PlayerDetail/Trade screen on mount

The existing WS subscribe/unsubscribe protocol is unchanged. The hub's server-side filter already works — only connections subscribed to a given instrument receive its Broadcast. Selective-sub reduces the subscription set from the client side.

Components displaying prices for non-subscribed instruments (market overview, leaderboard) receive stale-but-adequate prices from their REST initial load. Trade execution uses fill- at-shown-price with 2.5% slippage tolerance — a 10-second stale price on a browse page is within tolerance.

Alternatives considered

Option A (shard by instrument idx). Rejected: doesn't help the subscriber goroutine return faster — it still blocks on each Broadcast call waiting for worker channel sends. Pro (hot instrument isolation) is theoretical: in practice, price frames for a popular instrument arrive at 1 per 10s, same as any other; the bottleneck is fan-out volume, not per-instrument latency.

Option B (shard by connection). Same fundamental problem as Option A: subscriber goroutine still blocks per Broadcast call. Only helps if writes to different connections overlap in time, which they don't under the current sequential call model.

Selective-sub first (skip pool). Rejected for launch: selective-sub is a coordinated FE+BE change requiring store refactors, QA on subscription edge cases (reconnect, route change, position close during match), and a deployment window. Worker pool is a BE-only change with a clean internal boundary at hub.Broadcast. Pool ships first (launch gate), selective-sub ships within 7 days of launch as a performance improvement.

W=8 (lockdown minimum from prior reports). Confirmed insufficient for the 4-match Group Stage peak (drain 62.5ms vs 48ms inter-call window). W=16 is the correct launch value. The env var allows raising to 32 without a deploy if needed.

Blocking drop policy. Rejected: propagates backpressure to subscriber goroutine → Redis channel falls behind → missed price frames. A dropped write to a slow client is recoverable in 10 seconds; a missed Redis frame causes all clients to see a stale price.

Consequences

  • The subscriber goroutine is effectively non-blocking: it dispatches to worker channels in microseconds and returns to the Redis channel immediately.
  • At W=16 and 5K subscribe-all with 4 simultaneous matches, the broadcast pool operates well within its drain-time window (31.3ms vs 48ms inter-call).
  • Dead connections are detected within 45 seconds regardless of subscription activity, bounding ghost connection accumulation.
  • One new env var: WS_WORKER_COUNT=16. Required on the ws-server Container Apps only: ftl-prd-ws-cin and ftl-stg-ws-cin. The api-server (ftl-prd-api-cin) does not run the WS hub and must not receive this variable. The variable is provisioned as a GitHub Actions repository variable (per ws-191 fix) and injected by the deployment workflow into the ws-server apps only. Absent from ws-server apps as of 2026-06-08 [verified live]; T04 adds it to the workflow.
  • ws_broadcast_drops_total=0 is a hard load-test gate (T1 500-conn/120s, T2 2K-conn). Non-zero drops at sub-5K load indicates a worker thread bottleneck — investigate before proceeding to 5K test.
  • Phase 2 selective-sub is sequenced for Week-1 (post-launch). Until it ships, the pool handles 5K subscribe-all at launch load. After it ships, writes/s drop 71× and the pool is unnecessary capacity headroom.