Between 06:22 and 08:23 UTC on July 8, requests to our US error reporting API (api.honeybadger.io) experienced a highly elevated failure rate. During this window, a majority of requests to submit errors, events, and check-ins received 5xx responses or timed out. Error reports sent by client libraries without retry logic during this period were not received and cannot be recovered. The Honeybadger web application, dashboards, uptime monitoring, and our EU region were not affected.
What happened: Beginning at 06:18 UTC, we started receiving error reports at roughly triple our normal total API volume. The service tier that ingests this traffic was at its overnight capacity level and became saturated. Our autoscaling responded, but the metrics it relied on significantly under-measured the failed traffic, so capacity was added far too slowly — full recovery came at 08:23 UTC when scaling caught up and absorbed the load. Compounding the problem, PagerDuty alerts were not fired to wake up a human due to a misconfiguration.
What we've done: We've fixed the underlying scaling weakness: API capacity now also scales directly on load-balancer error rates (previously we scaled only on request rates, and that scaling doesn't respond quickly enough), and we've doubled the maximum capacity of the ingestion tier. These changes are already deployed to both US and EU regions. We're also evaluating adding adaptive per-key rate limiting to the ingestion service.
We're sorry for the disruption, and for any gaps this caused in your error data during the window.