Skip to content

spec: event-ingestion status: active last-updated: 2026-06-11 owners: [@amalkrsihna] code-refs: - ftl-backend/internal/sportmonks/ticker.go - ftl-backend/internal/sportmonks/events_feed.go - ftl-backend/internal/sportmonks/reconciler.go - ftl-backend/internal/sportmonks/postmatch.go - ftl-backend/internal/sportmonks/client.go - ftl-backend/internal/match/events.go - ftl-backend/migrations/000041_create_match_events.up.sql - ftl-backend/migrations/000042_create_event_bump_log.up.sql - ftl-backend/migrations/000043_create_event_recovery_tasks.up.sql decisions: [0019, 0023, 0024, 0025]

Football event ingestion & durability

TL;DR for an intern

When something happens in a real match — a goal, a card, a substitution — we read it from our data provider (Sportmonks) and use it to move player prices and fill the live "What's happening" feed. This spec describes how we make sure we never lose one of those events, even if a server restarts, Redis is wiped, or the provider hiccups, and how we make sure a single event never moves a price twice.

Glossary (defined before first use)

  • Sportmonks — the third-party football data API we poll for live matches.
  • Fixture — one match. Identified by a Sportmonks integer fixture_id.
  • Event — a discrete in-match happening (goal, card, sub, VAR, corner, …) with a provider id (provider_event_id).
  • Ticker — the background loop (RunMatchTicker) that polls live matches every 10 seconds and turns events/stats into prices.
  • micro_bump — the running price nudge a player's instrument accumulates from discrete events during a live match (see amm-pricing).
  • Redis — in-memory store used as a hot cache + pub/sub. Fast but volatile.
  • Postgres (PG) — durable SQL store; our source of truth.
  • Lease — a Redis SET-NX lock so only one api-server replica runs a given loop per tick (internal/lease).
  • Idempotent — safe to run more than once with the same result (no duplicate rows, no double price move).
  • inplay vs latest — two Sportmonks endpoints. livescores/latest is a ~10s delta (only recently-changed fixtures); livescores/inplay is a full snapshot of every in-play fixture.
  • events array vs timeline array — Sportmonks v3 splits live happenings across two parallel arrays on the fixture payload, each with its own integer ID space.
  • events (model_type=event) — the 12 canonical types: Goal, Own Goal, Penalty, Missed Penalty, Substitution, Yellowcard, Redcard, Yellow/Red card, VAR, Penalty Shootout Goal/Miss, Highlight.
  • timeline (model_type=timeline) — 7 lighter-weight log entries: Corner, Offside, Shot On Target, Shot Off Target, Woodwork, Delay Start, Delay End.
  • Both arrive in the same Sportmonks response when the timeline include is requested; without that include the timeline array is silently absent and offsides/corners never reach the system.

Formula / algorithm

INGEST (every 10s, lease:sportmonks-ticker, one replica):
  GET livescores/inplay?include=...;events.type;timeline;...
  for each live fixture (isolated by recover()):
    for each stream in (events, timeline):              # TWO STREAMS, SHARED FEED
      kind = 'events' | 'timeline'                       # namespaces the dedup set
      for each entry:
        if SADD seen_{kind}:{fixture} == 0: skip        # already processed (hot dedup)
        LPUSH match_events:{fixture}  (shared hot feed cache, TTL 24h, cap 500)
        INSERT match_events (...) ON CONFLICT DO NOTHING   # DURABLE source of truth
        if entry has a tradable player and a non-zero bump:
          INSERT event_bump_log (fixture, event, instrument) ON CONFLICT DO NOTHING RETURNING id
          if a row was inserted:                         # EXACTLY-ONCE gate
            HSET instrument:{id} micro_bump += capped(bump)

READ (GET /api/matches/:id/events):
  SELECT ... FROM match_events WHERE fixture_id=$1
    ORDER BY minute DESC, extra_minute DESC, id DESC LIMIT $n   # match-time order
  (fall back to Redis LIST if no durable rows yet / PG unavailable)

FALLBACK LADDER (no event permanently lost):
  L1 client: retry 3x exp-backoff+jitter on network/5xx; 429 -> RateLimitExhausted backoff
  L2 reconciler (60s, leased): re-fetch each live fixture's events, replay through INGEST
  L4 post-match: at FT, fetch events, persist (source=postmatch) as authoritative backstop
  L5 outage: event_recovery_tasks queue, drained with bounded retry
  L6 cold-restart: on boot, enqueue backfill task per live fixture
  L7 on-demand: ReplayFixture / scripts/replay-events <id> rebuild any fixture

The exactly-once guarantee rests on two Postgres uniqueness constraints: match_events (fixture_id, provider_event_id) (persist once) and event_bump_log (fixture_id, provider_event_id, instrument_id) (bump once). Both survive a Redis flush, a restart, and arbitrary replay.

Edge cases

  • Redis flush mid-match — seen_events and micro_bump are gone, so the next poll re-presents every event; match_events ON CONFLICT skips re-persist and event_bump_log skips re-bump, so no duplication. The reconciler/feed re-hydrate Redis from the durable timeline. (ADR-0023)
  • Missed poll window (deploy, GC pause, breaker open, lost lease) — inplay returns the full current list on the next successful poll, so nothing is permanently skipped; L2/L4 close any residue. (ADR-0023)
  • Out-of-order backfill — the reconciler may persist a minute-12 event after minute-80; the read orders by match time so it renders in the right place, not at the front. (ADR-0023)
  • Player-less events (corner, VAR) — persisted to the feed + timeline before the bump path; they appear in the UI but apply no price bump. (ADR-0023)
  • 429 / quota exhausted — GetAllPages returns empty and skips the poll while RateLimitExhausted() is true, rather than hammering a zero-quota key. (ADR-0023)
  • Team-propagation bumps — goal-driven team-wide bumps (ADR-0019) are NOT yet routed through event_bump_log; on a Redis-flush replay they could re-apply. Known follow-up. (ADR-0019, ADR-0023)
  • Best-effort durability — a transient PG error on persist/bump-gate is logged, not fatal: the live feed/price path continues (Redis remains the in-process guard) so a DB hiccup never stalls pricing. (ADR-0023)

Code references

  • ftl-backend/internal/sportmonks/ticker.go — PollLivescores (inplay source, lease gate), processFixture (recover), processEvents (dedup, persist, bump gate).
  • ftl-backend/internal/sportmonks/events_feed.go — pushMatchEvent (hot cache), persistMatchEvent (durable write), MatchEventDTO, normalizeEventType.
  • ftl-backend/internal/sportmonks/reconciler.go — EventReconciler (L2/L5/L6), reconcileFixture, ReplayFixture (L7), recovery-task drainer.
  • ftl-backend/internal/sportmonks/postmatch.go — ProcessCompletedFixture L4 event backfill.
  • ftl-backend/internal/sportmonks/client.go — Get/doRequest retry+backoff, RateLimitExhausted, circuit breaker.
  • ftl-backend/internal/match/events.go — (*Service).GetEvents durable ordered read + fallback.
  • ftl-backend/migrations/000041_create_match_events.up.sql, …/000042_create_event_bump_log.up.sql, …/000043_create_event_recovery_tasks.up.sql.
  • ADR-0019 — match-event propagation (team fallback, opponent drop, durable event_score).
  • ADR-0023 — durable, lossless event capture (this spec's primary decision).
  • ADR-0024 — per-event admin config (enable / per-event cap / category), applied cluster-wide via the admin-settings Redis pub/sub broadcaster.
  • ADR-0025 — per-event decay via a persistence split between the transient micro_bump and durable event_score channels.

How to verify

# Unit + DB integration (skips without Postgres; uses dev02 sandbox on 5433)
DATABASE_URL="postgres://ftl_admin:localdev@127.0.0.1:5433/ftl2026?sslmode=disable" \
  go test ./internal/sportmonks/ ./internal/match/

# On-demand replay of a fixture (idempotent — safe to re-run)
go run ./scripts/replay-events <fixtureID>

# Confirm the durable table fills + bumps fire exactly once during a live/replay match
psql "$DATABASE_URL" -c "SELECT count(*) FROM match_events WHERE fixture_id=<id>;"
psql "$DATABASE_URL" -c "SELECT count(*) FROM event_bump_log WHERE fixture_id=<id>;"

In the UI, the Stadium "What's happening" panel renders the timeline (newest match minute first); a goal/card/sub should appear within one poll even across an api-server restart.