We're good to go! We can boot instances again, so we now have replacement instances in place that serve our app and www domains. Our blog and docs domains are also back.
Posted Feb 28, 2017 - 15:14 PST
Update
The web application (app.honeybadger.io) is back online. If you are unable to connect:
1. Make sure you're using the app subdomain. The www subdomain is a separate site and is still down.
2. Try clearing your DNS cache. We had to update DNS to point to new servers.
We're not 100% out of the woods yet and probably won't be until AWS is back up. The new app server is a temporary solution and all of our static sites (www, docs and blog) are still down.
Posted Feb 28, 2017 - 14:32 PST
Update
Finally, a bit of good news!
Incoming error notices are being processed again. Notifications are being sent. The web app is still down for now, but we're working on that.
Posted Feb 28, 2017 - 14:13 PST
Update
AWS issues are ongoing. We're attempting to redeploy on alternate servers, which is harder than it should be since half of the web is down right now.
At this point our backlog queues have filled up and errors are being dropped.
Posted Feb 28, 2017 - 11:24 PST
Monitoring
The AWS issue is ongoing. It's looking like we may have to ride it out.
At the time of this writing, our API servers are still accepting requests and putting them in queue. Processing and delivering notifications will be delayed until we get the rest of our servers back up.
Posted Feb 28, 2017 - 10:12 PST
Identified
It appears that we may be caught up in a widespread AWS problem. Some of our app instances were unexpectedly terminated and failover failed. We're working on restarting the instances now.