Thursday December 22, 2016

First, we apologize again for the disruption this outage has caused. Here's a quick summary of what happened: Today, from 6:40am PST to about 10am PST Slack was down or slow for all customers. This was due to a slight decrease in database performance which exhausted all available capacity to answer requests and resulted in a large percentage of requests — signing in, sending messages, reacting, uploading files, and so on — failing.

Service was restored around 10am PST. None of your messages were lost but they remained queued for processing longer than usual. All messages sent during the incident have been processed now. We have taken several steps to address the issues and will be making these changes permanent in order to prevent similar incidents in the future. As always, we will continue to monitor the health of Slack and invite you to write to <a href="mailto:feedback@slack.com">feedback@slack.com</a> if you have problems. Thank you again for your patience and apologies for the disruption ahead of the holidays.

4:35 PM PST

Thank you for your patience. After Slack being down for about four hours for most of our customers, we are confident that the measures we've taken to restore service are functioning as expected. At this time everyone should be able to connect to and use Slack, including messaging, reacting, uploading, and all the other features that were impacted this morning. We are monitoring the changes we made to restore service closely and expect that at 4pm PST we will give the all-clear. We are extremely sorry for the interruption this has caused.

11:57 AM PST

We have observed significant improvement in the performance and availability of all parts of Slack in response to the changes we've made. There are still aspects of the service that are slower than they should be so we are not declaring this outage to be at an end just yet. We are still all-hands-on-deck as we continue to monitor the health of Slack. We are very sorry for the immense trouble this outage has caused.

11:07 AM PST

For the last 3 1/2 hours, Slack has been slow or down for nearly all customers. Those who were connected before this outage began may have remained connected and noticed some issues; most have been unable to connect at all. In general, every customer has been impacted in some way. Our operations team has been attempting to restore service and has confirmed some improvement recently. This means those who could not connect should be able to, though you may experience slowness until we have fully resolved things.

10:34 AM PST

After an internal technical change earlier this morning, a significant percentage of API requests were failing. This affected many Slack features, and led to partial-to-full unavailability of Slack to all of our customers. We have begun to fix things, and are monitoring the recovery. Our deepest apologies.

10:05 AM PST

We are still seeing issues with some Slack functionality and features. Editing messages, adding reactions, file uploads, bots, and many other aspects of the product are affected. We are in the midst of fixing things and will continue to provide updates. We hope to have the service back up and running shortly.

9:57 AM PST

Most users should be able to connect to slack at this time, but services for all users will still be degraded -- we are close to fixing this, hang tight!

9:48 AM PST

We have identified a resolution to the problem and are working to bring services back online.

9:42 AM PST

Slack is still degraded. It's taking time to resolve this incident because we're focusing on solutions which avoid data loss. We're continuing with our operational plan and will update. Our deepest apologies.

9:22 AM PST

Functionality and features such as editing messages, adding reactions, file uploads, and all the things that make Slack Slacky aren't working very well right now. We are operationalizing a plan to fix things and will continue to provide updates. We're so sorry.

8:55 AM PST

Functionality and features such as editing messages, adding reactions, file uploads, and all the things that make Slack Slacky aren't working very well right now. We are operationalizing a plan to fix things and will continue to provide updates. We're so sorry.

8:55 AM PST

It's not a very happy holiday for Slack Operations, but we're executing a plan to restore service. As of now, some API requests & file uploads are failing. Our deepest apologies.

8:44 AM PST

Slack teams may be experiencing slowness and/or an inability to login. We’re working hard to get things back to normal as quickly as possible and will continue to provide updates.

8:15 AM PST

Web and API requests are continuing to time out: meaning that for some, image & file uploads and bots and things may be impacted. Basically, all is not as it should be, but we’re working hard to get things back to normal as quickly as possible.

7:53 AM PST

Web and API requests are timing out and we are actively working on a solution. Image & file uploads and bots may be impacted.

7:27 AM PST

A number of users are experiencing problems connecting to Slack. The issue has been identified and we are working on a solution.

7:09 AM PST

Some services, such as link unfurling, may not be working. We are aware of the root cause and we are working on it

6:44 AM PST