Background Jobs

All goroutines below start inside cmd/api-server/main.go unless noted.

Sportmonks scheduler

Started by: go scheduler.Run(ctx) — only when SPORTMONKS_API_KEY is set.
Lives in: internal/sportmonks/

The scheduler polls Sportmonks every 2 seconds during live matches. Between matches it polls less frequently (using Sportmonks' own fixture-state transitions).

What it does per tick: - Fetches live fixture state and player events for every configured league. - Updates instrument live_base_price in Redis based on player performance (scoring bumps via sportmonks.SetEventBumps). - Writes match events to Redis events:{sportmonksId} list. - Publishes price:{instrumentId} for each changed price. - Stamps live_match_until = now+30s (epoch s) on each live-match player's instrument hash (field live_match_until). The noise generator reads this to determine whether to tick the player every cycle regardless of trade activity (ADR-0021). - On FT (full time): calls ProcessCompletedFixture which: 1. Fires marginReplayerAdapter.ReplayUsersAtFT (ADR-0006 pre-FT washouts). 2. Calls autoSellerAdapter.ExecuteSystemSell for every legacy open position. 3. Calls cfdCloserAdapter.CloseAtFTSnapshot for every open CFD position. 4. Sets HSET instrument:{id} frozen 1 during liquidation, clears after.

What breaks if not running: No live price updates during matches. FT auto-close of positions does not fire (users keep positions open past match end).

Squad syncer: sportmonks.NewSquadSyncer runs inside the scheduler. It syncs player rosters from Sportmonks into Postgres squad_players, triggering instrumentSvc.HydrateRedis to keep instrument state fresh; HydrateRedis now calls InvalidateCache after each sync so the cached instrument list (cache:instruments) does not serve stale names or prices.

Gotcha: Sportmonks 50-int filter cap

Sportmonks v3 rejects filters=*Types: lists with more than 50 integers (HTTP 400, error code 5010, "The types filter should be a comma seperated list of maximum 50 integers"). We hit this once on livescores/latest when the implicit lineupDetailTypes: filter accumulated 55 IDs — every poll silently failed, no price_ticks written, Stadium event feed dark. See ADR-0013. Resolution: the request is now sent without the filter; consumers filter client-side.

If you ever re-add a filters=...Types: value, keep it ≤ 50 integers, or better — generate it from the consumer code (scoring.go keys ∪ statDeltaTypes keys) and assert the cap at build time.

Noise generator

The NoiseGenerator (lives in internal/sportmonks/noise.go) runs alongside the scheduler goroutine. It publishes synthetic price:{instrumentId} ticks every ~2 s (configurable via NOISE_INTERVAL_MS, default 2000 ms) to keep the chart moving between real Sportmonks events. Key behaviours after ADR-0021:

1. Instruments in a live match (live_match_until > now) publish every cycle regardless of trade activity.
2. The noise offset carries across pattern windows — loadOrSelectPattern seeds new patterns with the persisted offset rather than resetting to 0. Global mean-reversion applies each tick: offset -= offset / tauTicks where tauTicks = NOISE_MEANREVERT_MINUTES × (60000 / intervalMs). Default NOISE_MEANREVERT_MINUTES = 10; set to 0 to disable reversion.
3. NOISE_DELTA_SKIP_PCT (default 0) means every tick whose rounded price differs from the last published price is broadcast — the prior default was 1%.
4. noiseOffsetClamp is ±0.08 (fraction of AMM base price).

Environment variables for the noise generator and trade slippage:

Variable	Default	Notes
NOISE_INTERVAL_MS	2000	Tick interval for the noise generator in milliseconds.
NOISE_MEANREVERT_MINUTES	10	Mean-reversion time constant. 0 disables reversion. (ADR-0021, new)
NOISE_DELTA_SKIP_PCT	0	Minimum fractional price-change required to publish a tick. Default 0 = publish every changed tick. Was 0.01 (1%).
TRADE_SLIPPAGE_TOLERANCE_PCT	0.025	CFD slippage tolerance (2.5%). Was 0.005 (0.5%). (ADR-0020)

Cost note: NOISE_DELTA_SKIP_PCT=0 combined with live_match_until gating means every live-match player publishes a price tick approximately every 2 s during an active match. During a live fixture with 22+ players this substantially increases Redis pub/sub operations compared to the previous activity-gated behaviour. Assess fan-out cost before disabling the skip threshold in production at scale.

Weekly spinner

Started by: go runWeeklySpinner(ctx, spinnerSvc).
Lives in: internal/spinner/

Runs once at boot (to catch any missed Monday trigger after a restart) then every Monday at 00:05 UTC.

Calls spinnerSvc.SpinFor(ctx, time.Now()). Uses ON CONFLICT DO NOTHING in Postgres so multiple api-server replicas spinning simultaneously produce exactly one winner row.

What breaks if not running: No weekly tiebreaker winner selected. Leaderboard still functions; the spinner result is a display feature.

Margin reconnect listener

Started by: go runMarginReconnectListener(ctx, redisClient.RDB(), marginSvc).
Lives in: cmd/api-server/main.go

Subscribes to the Redis user:reconnect channel. ws-server publishes a message each time a user transitions from 0 to 1 active connections.

For each received userId, spawns a goroutine calling marginSvc.ReplayLastSeen(ctx, uid). The replay reads the price_ticks buffer since last_seen:{userId} and closes any positions that breached their margin or SL/TP at the breach price — not the current price.

What breaks if not running: Offline users' breached positions accumulate until the 5-min backstop sweeper catches them. No data loss, but breach-price settlement is delayed.

Margin backstop sweeper

Started by: go runMarginBackstopSweeper(ctx, pool, redisClient.RDB(), marginSvc).
Interval: Every 5 minutes.
Redis lease: lease:margin-backstop (TTL 30s, acquire via SETNX).

Queries SELECT DISTINCT user_id FROM cfd_positions WHERE closed_at IS NULL. For each user, calls marginSvc.ReplayLastSeen. Logs total users swept and washouts fired.

Only one api-server replica runs the sweep at a time; others skip the tick if they cannot acquire the lease.

What breaks if not running: Stranded offline users whose WebSocket never reconnects accumulate undetected washouts. The reconnect listener handles online users; this sweeper handles the permanently-offline case.

StopWatch (SL/TP enforcement)

Started by: go positionsStopWatch.Run(ctx).
Lives in: internal/positions/stopwatch.go
Redis lease: lease:stop-watcher.

Subscribes to price:* pub/sub. On each tick: 1. Reads all open positions from Postgres (cached in memory, refreshed every 5s). 2. For any position where the new price breaches stop_loss or take_profit, calls position_close.lua. 3. Also calls marginTickAdapter.EvaluateAndApplyForTick(ctx, userID) for the margin wash-out check.

Single-instance across replicas via the Redis lease.

What breaks if not running: SL/TP orders are not enforced in real time. The 5-min backstop sweeper provides a fallback, but with up to 5 minutes of drift.

Bot engine

Started by: go botEngine.Start(ctx) — only when BOT_ENABLED=1.
Lives in: internal/bot/

Simulates synthetic traders (bot personas) buying and selling instruments. Each persona has a wallet funded from BOT_WALLET. Bots provide liquidity when no real users are active.

The scheduler starts the bot engine when a match goes live. When SPORTMONKS_API_KEY is not set (dev mode without live matches), the engine starts immediately at boot.

What breaks if not running: No synthetic liquidity. Price does not move without real user trades.

Embedded flusher

Started by: go flushSvc.Run(ctx) — only when FLUSHER_EMBEDDED=1.
Lives in: internal/flusher/

Runs the flusher loop and both outbox drainers inside the api-server process. Used in single-binary local dev. In production, the standalone cmd/flusher binary is deployed instead.

See Services — flusher for the drain details.

Google JWKS refresh

Started by: go googleVerifier.StartRefresh(ctx).
Interval: Every 6 hours.

Refetches https://www.googleapis.com/oauth2/v3/certs into the in-memory keys map (RW mutex protected). A failure is logged but does not crash the process; the previous keys continue to work until a Google key rotation causes a verification failure, at which point the next scheduled refresh corrects it.

What breaks if not running: After a Google key rotation (rare; Google rotates keys roughly weekly), new Google logins fail until the next refresh.

Known issues (Sportmonks upstream)

The following are upstream data gaps, not backend bugs:

• Goal and card event feed empty for some fixtures. The Sportmonks feed omits goal/card events for certain fixtures; the events:{sportmonksId} Redis list is populated but contains no goal or card entries for those games.
• Per-player long-ball / aerial-won / tackle stats absent for U18 competitions. Only fixture-level team-aggregate stats are available; per-player stat deltas for these categories do not arrive from the Sportmonks API for U18 leagues.
• Kazakhstan U18 squad returns 0 players. Instruments are created from the fixture list but the squad endpoint returns an empty roster for this team on the current data plan.