Now that we've conducted our postmortem and root cause analysis, we know the event summarized below started at 6:17 a.m. PDT (7/11/18).
4:48 PM PST
Thank you for your patience. Everything is back up and running and we wanted to give you a summary of what happened.
On June 27th (yesterday) between 6:33 a.m. and 9:49 a.m. PDT Slack experienced an outage where people could not connect to their workspaces. The network problems were caused by a bug included in an offline batch process of data, which resulted in unexpected network spikes and led all of our customers to become disconnected and unable to reconnect.
Once we identified the problem, we restricted new connections and provisioned extra capacity. At 9:24 a.m. PST, production was healthy enough to remove restrictions and by 9:44 a.m. PST, all customers had reconnected to Slack once again.
Our reliability as a service is core to our commitment to our customers, and we are deeply sorry for the disruption. Thank you again for your patience.
3:27 PM PST
We're happy to report that workspaces should be able to connect again, as we've isolated the problem. Some folks may need to refresh (Ctrl + R or Cmd + R). If you're still experiencing issues, please drop us a line at firstname.lastname@example.org.
9:49 AM PST
Our efforts to isolate the problem and resume connections are ongoing. Your patience through this is greatly appreciated.
9:33 AM PST
We're getting closer to a solution, but we aren't out of the woods yet.
9:03 AM PST
We really appreciate you sticking with us. We know this is a big disruption to your day, and we've got all eyes on this issue to get us back up to normal.
8:33 AM PST
We have no new information to share just yet, but we're continuing our efforts. Your patience is truly appreciated.
8:03 AM PST
We are continuing to work on fixing the connection problems that have been impacting folks. We hope to have the issue fully resolved as soon as possible.
7:33 AM PST
Our team is still looking into the cause of the connectivity issues, and we'll continue to update you on our progress.
7:03 AM PST
We've received word that all workspaces are having troubles connecting to Slack. We're currently investigating the issue, and will have updates shortly.
6:33 AM PST