resolved

Webhook delivery delays and temporary API slowdown

An overly aggressive loop-protection rule repeatedly paused webhook processing for affected workspaces, delaying some deliveries by several hours. A subsequent recovery change incorrectly rejected some valid events, while backlog processing increased database load and slowed API requests. We rolled back the new enforcement, reduced replay pressure and deployed corrections. API response times recovered, but webhook delays remain for affected workspaces. We are validating a permanent fix and reviewing affected events for safe recovery.

Started: Mon, 02 Nov 2026 13:00:00 GMT

Updates

1. resolvedTue, 03 Nov 2026 18:26:04 GMT

   The fix is deployed and the affected webhook backlog has cleared. Subsequent checks confirm processing is caught up and API response times are within expected thresholds. This incident is resolved. We will continue monitoring, and the review of historical delivery exceptions will continue separately.

2. monitoringTue, 03 Nov 2026 14:17:18 GMT

   We are deploying a fix to improve webhook delivery reliability and recovery of delayed events. We are monitoring delivery progress, remaining delays and API response times throughout the rollout. Some webhook deliveries may remain delayed during deployment and recovery. We will provide another update once we have verified the fix and recovery progress.

3. identifiedTue, 03 Nov 2026 11:33:27 GMT

   An overly aggressive loop-protection rule repeatedly paused webhook processing for affected workspaces, delaying some deliveries by several hours. A subsequent recovery change incorrectly rejected some valid events, while backlog processing increased database load and slowed API requests. We rolled back the new enforcement, reduced replay pressure and deployed corrections. API response times recovered, but webhook delays remain for affected workspaces. We are validating a permanent fix and reviewing affected events for safe recovery.