At-risk alert never arrived and mushrooms ran out — inventory-predict cron claim leak (PR #606 CCC F1)

Aitor (47) owns Pintxos del Mercado 38-cover Basque pintxos bar Bilbao Casco Viejo Calle del Perro two minutes from Ribera Market. 12 years on place weekend tourist flow heavy. thMenu Pro+ 14 months daily 09:30 ingredient-reorder-alert + 07:00 inventory-predict cron Apple Watch at-risk push notifications. Friday 08:30 head chef Iker "Aitor cèpe mushroom for seafood pintxos shipment hasn t arrived." Aitor surprised no inventory push last 3 days normally 2-3 stock alerts/day. Engineering cron_idempotency_claims query Aitor restaurant_id last 7 days: Monday 05-19 07:00 UTC inventory-predict:aitor:2026-05-19 INSERTed claimed_at set released_at NULL. Tuesday-Wednesday-Thursday-Friday 07:00 fresh claim attempt existing row changes=0 cron skip. 5 days consecutive inventory-predict skip Apple Watch never buzzed. Wrong theory: Cloudflare cron didn t fire? Workers tail logs Monday 07:00 [BEACON:cron_fired] log entry exists Cron error AbortError timeout 5sec aiResponse fetch throw runCronSafe try/catch swallow log+return but claim NOT released. cron_idempotency_claims pattern at-least-once dedup INSERT OR IGNORE claim winner side-effect fire pattern works ONLY when side-effect succeeds. res.ok=false handled — release branch existing code catches res.ok=false. fetch throw (AbortSignal.timeout network blip DNS fail) outer try/catch swallows release branch never triggered claim held 24+h. Pattern: claim acquisition before side-effect call → fetch attempt → on success predictions + push fan-out / on res.ok=false releaseClaim + return / on throw control flow throw branch function exits with exception outer runCronSafe catches log+return RELEASECLAIM NEVER CALLED. Aitor case 5 consecutive days AI inference timeout 5 consecutive claim leaks. **PR #606 batch CCC F1** fix sober: side-effect call (fetch AI inference) wrapped in inner try/catch: `let aiResponse = null; try { aiResponse = await fetch(...); } catch (err) { await releaseClaim(env, claimKey); throw err; }`. Throw branch calls releaseClaim claim reverts re-throw outer runCronSafe logs next tick claim available cron fires again. Bonus Sentry [BEACON:inventory_predict_fetch_throw] beacon 5+ throw/hour threshold PagerDuty sustained throw upstream AI service problem. Backfill: Aitor manual inventory-predict reset+replay 5-day leaked claims DELETE manual cron fire AI current state cèpe-mushroom+4 other at-risk pushes Apple Watch. Platform-wide audit released_at IS NULL AND claimed_at < now-24h orphan leaked claims: **7 affected restaurants 23 leaked-day total**. Reset+replay each. Proactive email + 1-month Pro tier credit. Aitor Hall of Fame "helped harden cron claim symmetry". Mert Ayvalık Cunda "Ayvalık Köfte + Deniz" 65-cover Aegean seafood version with same flow mushroom. Pattern: **in claim-before-side-effect pattern, side-effect call (fetch r2.put kv.put getEnv) must be wrapped in inner try/catch + releaseClaim called on throw + re-throw. Outer runCronSafe only catches synchronous failures — if release isn t called on every path, claim leaks 24+h → cron skips → side-effect (alert/notification) never fires.** Implementation checklist: (1) acquire claim before side-effect; (2) side-effect inner try/catch; (3) throw releaseClaim + re-throw; (4) symmetric coverage every await external; (5) test simulate AbortSignal.timeout assert release+retry; (6) Sentry [BEACON:cron_fetch_throw] alert 5+/hour; (7) quarterly orphan-claim audit released_at NULL claimed_at < now-24h; (8) sibling cron sweep postback dispatcher cache-purge image-proxy custom-domain-reverify DoH.

Related articles