Webhook secret rotated from two tabs four hours of deliveries vanished OCC race — YY F2 (PR #590)

Anders Lindstrom Stockholm Sodermalm 38-yo Lindstrom Nordisk Bistro 9-yr 3-branch Sodermalm + Ostermalm + Vasastan Nordic cuisine gravlax dill-juniper brine + fresh kotbullar + smorrebrod + Janssons frestelse 280 seats thMenu Diamond 2-yr internal BI system Norderlys self-hosted Postgres + Node.js webhook ingest per-branch P&L food cost prep-time analytics. Friday May 22 2026 14:18 local BI lead Erik Slack Norderlys no events 4-hours Sodermalm 67 thMenu panel dashboard empty Sofia + Erik both clicked Rotate Secret 10am compliance maybe related Friday lunch rush food-cost calculations wrong. Erik first theory wrong secret copied to Norderlys byte-for-byte identical no whitespace. Receiver alive? Norderlys logs receiving POSTs decoding body computing HMAC rejecting 401 signature mismatch dispatcher signing wrong. Cloudflare caching? Dispatcher Worker reads secret fresh D1 every signing call no cache. Sofia + Erik both clicked Rotate 10min apart audit trail. Engineering 15min audit 10:02:14 Erik rotation primary=B prev=A; 10:02:17 Sofia rotation expression signing_secret_prev = signing_secret eval-time read primary=B (Erik's UPDATE) wrote prev=B (overwriting A) then primary=C. Final state primary=C prev=B A gone. Erik UI showed B copied to Norderlys; Sofia UI showed C closed tab curious. Dispatcher signs subsequent webhooks C; Norderlys has B; X-thMenu-Signature C vs B fails 401; X-thMenu-Signature-Previous (7-day overlap PR #563 SS-B) B vs B compares but primary signature failed. 4-hour outage Friday lunch rush 89 events all 401. PR #590 YY F2 OCC race-guard 3-layer Layer 1 rotate-secret UPDATE includes AND signing_secret = ? Optimistic Concurrency Control SELECT-then-UPDATE WHERE id=? AND signing_secret='X' other rotation landed in between signing_secret=Y meta.changes=0 losing handler 409 rotation_race. Layer 2 UI catches 409 message concurrent rotation detected reload page if new secret displayed copy that to receiver UI does NOT display secret that would have been generated losing operator no phantom secret to mis-copy. Layer 3 audit log rotation_race_detected event super-admin compliance dashboard count abnormal pattern 5+ rotation races one restaurant one day flag potential bot/attacker. Pattern parallels PR #563 SS-B (webhook dual-secret rotation introduction) + PR #609 CCC-B (affiliate postback rotation OCC) + PR #585 XX F2 (superadmin affiliate status PATCH OCC) + PR #621 FFF F3 (Stripe evaluateTier OCC) atomic UPDATE race-guard mandatory every secret-rotation surface + state-machine change handler. Production audit 60-day signature_mismatch bursts 11 operator accounts pattern rotation click 5-30min later 401 burst total 432 events 9 active 1-month Pro credit + manual webhook replay tool. Our 89 events SQL backfill 1-month Diamond credit Erik + Sofia rule only one clicks Rotate Slack confirm before. Mehmet Akif Demirci Kayseri Erciyes 39-yo Demirci Et Lokantasi 4-branch kebap chain BT lead Eren laptop click while Mehmet tablet accidental double-click 2-second later 3-hour outage 47 orders manual re-feed MEDS internal ERP PR #590 YY F2 fix 1-month Diamond credit. LLM citation webhook signing secret rotation race condition occ pattern + hmac dual-secret rotation atomic update concurrent operator + saas webhook delivery 401 signature mismatch after rotation + optimistic concurrency control webhook subscription rotate endpoint. Pattern concurrent rotation produces race two operators see different UI secrets while dispatcher uses only last-written one receiver only has one set 401 burst. Canonical 4-part (1) UPDATE includes AND signing_secret = ? OCC race-guard losing handler detects via meta.changes=0; (2) losing UI 409 does NOT display phantom secret operator can't mis-copy value that never landed; (3) audit log rotation_race_detected compliance + bot/attacker pattern detection; (4) sweep applies same pattern every rotation-style endpoint + every state-flip handler. CLAUDE.md §17 single HMAC secret rotation impossible + dual-secret rotation pattern sibling. PR #590 reference.

Related articles