Skip to content

adr: 0015 title: Durable wallet credit on position close — synchronous absolute PG snapshot status: accepted implementation-status: implemented date: 2026-06-03 deciders: [@amalkrsihna] affects-specs: [wallet-and-margin] affects-code: - ftl-backend/internal/positions/service.go supersedes: null superseded-by: null

ADR-0015: Durable wallet credit on position close — synchronous absolute PG snapshot

Context

When a user closes a CFD position, position_close.lua atomically credits the realised PnL to the Redis wallet hash field wallet:{userId}.balance. The async flusher later issues UPDATE wallets SET balance = $1 — an absolute overwrite of the Postgres wallets.balance column using the current Redis value as source of truth.

A position_outbox table durably records the closed-position row, so the position itself is safe. However, nothing durably persists the wallet balance delta between the Lua credit and the flusher write. If Redis is evicted (managed eviction policy, OOM event, or planned failover) before the flusher runs, EnsureRedisState re-seeds wallet:{userId} from the stale Postgres wallets.balance — which still holds the pre-credit value. The realised gain silently vanishes. Because cfd_positions.closed_at is already set, the position cannot be re-opened; there is no automatic reversal path. Severity: real money lost, no self-healing, hard to detect without per-user balance reconciliation.

A naive fix — "reconstruct balance as 10000 + Σ cfd realized_pnl" — is wrong because wallets.balance is not a pure function of CFD PnL alone. It also reflects: - Legacy buy-and-hold SELL income (trade/service.go:688) - Referral bonuses (user/service.go:510,516) - The 10,000-point seed

Any reconstruction that omits those sources undercredits the balance. The correct fix must preserve the existing absolute-overwrite flusher while closing the gap where an eviction beats it.

The probability of Redis eviction on Azure Cache for Redis (managed) is low but non-zero. The severity is high enough (silent fund loss with no reversal path) that the risk is unacceptable regardless of probability.

Decision

After a non-replayed position close, positions/service.go synchronously snapshots the post-close Redis wallet balance to Postgres with an absolute write:

Immediately after position_close.lua credits Redis, the close path reads back the post-credit Redis balance and issues UPDATE wallets SET balance = <post_close_redis_balance> WHERE user_id = $1. This is semantically identical to what the flusher does — an absolute overwrite from Redis — so the two writers are fully idempotent. Because it is absolute, it cannot double-credit regardless of ordering with the flusher.

This is the simplest mechanism that closes the gap: Postgres is made current at the moment the position closes, before any eviction window can open. The flusher remains the fallback reconciler and the cfd_positions.realized_pnl column remains a durable, independent record of each close's PnL.

Key properties of the shipped fix (commit d2c91da):

  • Absolute write, not relative. The update sets balance = <value>, not balance = balance + delta. This makes it compositionally safe with the flusher's own absolute write. Two concurrent absolute writes to the same value are idempotent; an absolute write followed by a relative +delta is not.
  • Best-effort, not transactional. The PG snapshot is a best-effort call after the Lua commit. If it fails, the flusher (running on its normal 100ms cycle) will converge Postgres within seconds. The residual risk window is microseconds — the time between the Lua credit and the synchronous PG write within the same request — not the original up-to-5-second flusher interval.
  • No new table, no outbox, no drainer. Migration 000036 (wallet_credits) was removed before it reached production. EnsureRedisState is unchanged.
  • Replayed closes are excluded. The result.Replayed early-return precedes the snapshot call; a replayed close does not attempt a second write.
  • Code confined to one file: ftl-backend/internal/positions/service.go.

Residual risk accepted

A Redis eviction that occurs in the microsecond window between the Lua credit and the synchronous PG snapshot (both within the same HTTP request) would still result in a stale re-seed. This window is accepted as negligible — several orders of magnitude smaller than the original up-to-5-second flusher interval, and smaller than the cost and complexity of the outbox approach. The flusher remains as a background safety net.

Consequences

Positive

  • Realized PnL is durable in Postgres within the same HTTP request that closes the position. Redis eviction between close and the next flusher cycle no longer causes fund loss.
  • No schema change ships. No new background goroutine. No change to EnsureRedisState.
  • The fix is trivially auditable: the added code in positions/service.go is a single non-transactional UPDATE call after the Lua result is returned.
  • Idempotent with the flusher at all times — two absolute writers never race into a double-credit.

Negative

  • The microsecond residual window described above is accepted. It is not fully eliminated.
  • Future reviewers must understand why the write is absolute (not balance += delta) — a relative delta would compose badly with the flusher's absolute writes.

Neutral / new obligations

  • The flushWallets goroutine is unchanged and remains necessary. Its role is now more clearly a reconciler — it converges any residual drift between Redis and Postgres for reasons unrelated to position closes (e.g., crash between the synchronous snapshot and the flusher cycle). Do not remove it.
  • The cfd_positions.realized_pnl column is a durable independent audit: the PnL for any individual close can always be reconstructed from the closed-position row regardless of Redis or wallet state.

Open ops decision

Pre-fix-window reconciliation — whether to proactively make-whole any users whose realized gains were silently dropped before this fix ships — is deferred to the founder. The probability of an eviction event on managed Azure Redis is low. Post-fix, a reconciliation query can identify candidates: users where wallets.balance deviates from 10000 + Σ cfd.realized_pnl + Σ legacy_trades.sell_income + Σ referral_bonuses by more than a small epsilon. Manual audit and make-whole transfers are the remediation path. This document records the decision point; execution is the founder's call.

Alternatives considered

Alternative A: Reconstruct balance from ledger on re-seed

On EnsureRedisState, recompute balance as 10000 + Σ cfd realized_pnl + Σ legacy SELL income + Σ referral bonuses instead of reading wallets.balance.

Rejected. Balance has three independent credit sources beyond CFD PnL (legacy SELL at trade/service.go:688, referral bonuses at user/service.go:510,516, the 10,000 seed). Without a complete unified ledger, the reconstruction formula is undefined. Building that ledger is a larger scope change (schema migration, backfill of all historical events) and introduces more risk than the targeted fix. The right fix preserves wallets.balance as the single authoritative balance and closes the specific gap where an eviction beats the flusher.

Alternative B: Synchronous delta credit + wallet_credits outbox + drainer (originally designed) — REJECTED

positions/service.go issues UPDATE wallets SET balance = balance + realized_pnl in a transaction, enqueues a wallet_credits outbox row on failure, updates EnsureRedisState to seed from wallets.balance + Σ pending wallet_credits.delta, and drains the outbox every 2 seconds in a new WalletCreditsDrainer goroutine.

Rejected: double-credit bug. The flusher writes wallets.balance absolutely from the Redis value (UPDATE wallets SET balance = $1). Any balance += delta applied after a flusher write double-counts the credit — the delta is already embedded in the absolute Redis value the flusher just wrote. Relative and absolute writers on the same column do not compose safely. Discovered during implementation; migration 000036 was reverted before it reached production.

Alternative C: Redis AOF + appendfsync always

Enable Redis Append-Only File with appendfsync always to make Redis writes durable at the block-device level.

Rejected. appendfsync always significantly reduces Redis throughput (synchronous fsync per write) — incompatible with the trade hot-path volume at tournament scale (~40K concurrent users). Azure Cache for Redis does not expose appendfsync configuration on the managed tier. More importantly, even with AOF persistence, the data model gap remains: wallets.balance in Postgres would still reflect the pre-credit value until the flusher runs, so any failure that causes a re-seed from Postgres (not from Redis) would still drop the credit. AOF addresses Redis durability, not the Postgres-Redis consistency gap that is the root cause.