Skip to content

Real-Time Updates

Two classes of updates reach browsers in real time: price broadcasts (binary msgpack, one per instrument per tick) and portfolio frames (JSON, one per user per trade or position change). Both travel the same path: Redis pub/sub backplane → ws-server subscriber goroutine → hub fan-out → WebSocket frame.

WebSocket Token

The ws-server does not hold a Postgres connection. It validates connections using a short-lived JWT signed by the api-server's Ed25519 key.

  1. Client POSTs to POST /api/ws/token on the api-server with its access JWT in the Authorization header.
  2. The api-server issues a ws-token signed with JWTWSExpiry (60-second TTL).
  3. Client passes the ws-token as a query parameter: wss://api.ftl2026.com/ws/prices?token=<ws-token>.
  4. The ws-server middleware (cmd/ws-server/main.go) calls jwtValidator.Validate(token, auth.TypWS) before the upgrade. A missing or invalid token returns HTTP 401 before the WebSocket handshake completes.

The 60-second TTL limits the window in which a leaked token can open a connection without a valid session.

Hub

File: ftl-backend/internal/ws/hub.go

Each ws-server replica instantiates one Hub with ws.NewHub(20000). The limit is physical connections, not subscription slots. IsFull() gates on physConns >= maxConns.

The hub maintains three maps under a single sync.RWMutex:

Map Key Value
subscriptions uint16 instrument index set of *websocket.Conn
portfolioSubs string userId set of *websocket.Conn
connUserID *websocket.Conn string userId

Subscription uses a numeric instrument index (uint16), not the full UUID string. The compact index fits in 2 bytes of the msgpack frame. Clients can send either {"action":"subscribe","idx":42} or {"action":"subscribe","instrumentId":"<uuid>"}. For the string form, the subscriber looks up the index via subscriber.LookupIdx, which reads from a instrument_idx hash in Redis.

Decision: numeric index over UUID string in wire frames — reduces each PriceUpdate from ~36 bytes (UUID string) to 2 bytes (uint16), keeping the total msgpack payload at 25 bytes and the wire size at 95 bytes per frame.

Price Broadcast Pipeline

sequenceDiagram
    participant Ticker as api-server (Sportmonks poller)
    participant Lua as Redis Lua script
    participant RD as Redis pub/sub
    participant Sub as ws-server Subscriber goroutine
    participant Hub as ws-server Hub
    participant Conn as WebSocket connections

    Note over Ticker: Every 2s during live match
    Ticker->>RD: HSET instrument:{id} live_base_price <new>
    Ticker->>Lua: EVALSHA (price update path)
    Lua->>RD: PUBLISH price:{instrumentId} JSON{price, netImbalance, source}

    Note over Lua,RD: Also fires from every trade/position open/close

    RD->>Sub: deliver message on price:{instrumentId}
    Sub->>Sub: decode JSON, resolve idx via instrument_idx hash
    Sub->>Hub: hub.Broadcast(&PriceUpdate{InstrumentIdx, Price, Source})
    Hub->>Hub: msgpack.Marshal(PriceUpdate) — 25 bytes
    Hub->>Hub: RLock, collect conns for idx
    loop each subscribed connection
        Hub->>Conn: WriteMessage(BinaryMessage, data) with 100ms deadline
    end

PriceUpdate wire format

Defined in ftl-backend/internal/ws/hub.go:

type PriceUpdate struct {
    InstrumentIdx uint16  `msgpack:"i"` // compact instrument index
    Price         float64 `msgpack:"p"` // current market price
    Source        uint8   `msgpack:"s"` // SourceLive=0, SourceTrade=1, SourcePostmatch=2, SourceReplay=3, SourceSynthetic=4
}

The struct marshals to 25 bytes msgpack. With WebSocket frame header, TLS 1.3 record, and TCP/IP headers, each frame is 95 bytes on the wire. At 1 message/2 seconds for 40K chart viewers, total outbound bandwidth is ~1.9 MB/s.

permessage-deflate is off. The payload is already binary and too small (25 bytes) for deflate to save meaningful space.

A write timeout of 100ms bounds each WriteMessage call. If a connection cannot accept the frame within 100ms, it is evicted: hub.UnsubscribeAll(c) + hub.RemoveConn(c) run in a separate goroutine.

Portfolio Frames

Portfolio pushes are text JSON (not binary msgpack). They are sent after every trade or position change.

The Lua script publishes to portfolio:{userId} inside the same atomic execution as the trade. The ws-server Subscriber goroutine routes these to hub.BroadcastToUser(userID, payload). The hub looks up portfolioSubs[userID] and writes the raw JSON bytes to each connection with the same 100ms write deadline.

The client subscribes to its own portfolio channel by sending: 

{"action": "subscribe_portfolio"}

The userID is taken from the upgrade-time JWT claims — the client does not supply it in the message.

Reconnect and Offline Window Replay

ADR-0006 defines the offline-window replay protocol.

On last disconnect (1 → 0 transition):

The hub fires onDisconnect(userID). The ws-server hook writes: 

SET last_seen:{userId} <epochMs> EX 86400

On first reconnect (0 → 1 transition):

The hub fires onReconnect(userID). The ws-server hook publishes: 

PUBLISH user:reconnect <userId>

The api-server's runMarginReconnectListener goroutine subscribes to user:reconnect. On each message it spawns a goroutine that calls margin.Service.ReplayLastSeen(ctx, userId). That function reads last_seen:{userId}, queries price_ticks for the window [last_seen, now], and applies any margin washouts that should have fired while the user was offline at their breach prices — not the current live price.

A 5-minute backstop sweeper (runMarginBackstopSweeper) handles users whose browser closed without a clean disconnect. It acquires a lease:margin-backstop Redis key (30s TTL, SetNX) so only one api-server replica runs the sweep at a time.

For price-tick replay on WS reconnect (missed chart updates), the client sends {lastSequence: N} after reconnecting. The server replays ticks from price_ticks since that sequence. If the replica died, the client re-subscribes to a fresh replica and receives current state.

Scaling Notes

Three ws-server replicas each hold up to 20,000 connections (60K theoretical capacity, budgeted at 50K active). Each replica subscribes independently to all price:* Redis pub/sub channels. A single PUBLISH from a Lua script reaches all replicas; each replica fans out independently to its local connections.

This means a single trade causes exactly one Redis PUBLISH but N_replicas × N_subscribers WriteMessage calls — an intentional tradeoff. The alternative (routing through a single replica) would require a distributed routing table.

Sticky session affinity (Container Apps ingress cookie) keeps reconnecting clients on the same replica. If that replica has crashed, the load balancer routes to any alive replica and the client re-subscribes.