Friday June 28, 2019

Issue Summary:

On June 28, 2019 at 4:30 a.m. PDT some of our servers became unavailable, causing degraded performance in our job processing system. This resulted in delays or errors with features such notifications, unfurls, and message posting.

During this time, approximately 10-25% of jobs resulted in errors or failure. By 10:00 a.m. PDT, we fully restored message delivery and reduced the error rates to less than 5% as the team continued to work on a full recovery.

At 1:05 p.m. PDT, a separate issue increased server load and dropped a large number of user connections. Reconnection attempts further increased the server load, slowing down customer reconnection. Server capacity was freed up eventually, enabling all customers to reconnect by 1:36 p.m. PDT.

Full service restoration was completed by 7:20 p.m. PDT. During this period, customers faced delays or failure with a number of features including file uploads, notifications, search indexing, link unfurls, and reminders.

Now that service has been restored, the response team is continuing their investigation and working to calculate service interruption time as soon as possible. We’re also working on preventive measures to ensure that this doesn’t happen again in the future. If you’re still running into any issues, please reach out to us at feedback@slack.com.

7:58 PM PST

A fix has been rolled out and all features should be working normally. We are immensely sorry for the disruption to your work day and are continuing the work to make sure this doesn't happen again.

If you are still seeing any issues, please don't hesitate to drop us a line via our help page or feedback@slack.com — we're here to help.

8:15 PM PST

We're seeing positive signs with some changes being rolled out, and are continuing to work to get things back to 100%. We really appreciate your patience.

7:27 PM PST

Customers may see degraded functionality for several other features, including notifications, reminders and workspace member lists — we're working to restore that. File uploads should now work after a refresh (Cmd/Ctrl + R).

6:14 PM PST

We’re still working hard to restore file upload functionality. Thank you for your continued patience.

4:54 PM PST

File uploads are still not working for some users but refreshing Slack (Cmd/Ctrl + R) may restore functionality.

3:56 PM PST

We don't have any updates right now. Refreshing Slack (Cmd/Ctrl + R) should restore connectivity but we’re still working on file uploads.

3:25 PM PST

We expect users to be able to connect to Slack after a refresh (Cmd/Ctrl + R) but we are still experiencing some issues with file uploads.

2:54 PM PST

We are still seeing reports from many users unable to connect to Slack. We truly appreciate your patience while we monitor the situation.

2:24 PM PST

Some users have reported problems loading Slack. We'll continue to update you on our progress.

1:54 PM PST

Some users have reported intermittent connectivity issues. We remain hard at work to come to a full resolution.

1:24 PM PST

Still no news to share on a resolution, but please continue to let us know if you're running into any trouble after reloading. We'll return with more updates soon.

12:51 PM PST

We're still working on getting things completely fixed up, but for now reloading should help. If this doesn't lead to improvements, though, drop us a line at feedback@slack.com.

12:21 PM PST

We’re seeing that functionality may be restored after reloading (Cmd/Ctrl + R). If you're still running into trouble after giving that a try, please let us know at feedback@slack.com.

11:51 AM PST

No news to share just yet, but we remain hard at work to get everything running smoothly again. We appreciate your patience in the meantime.

11:34 AM PST

We're seeing some signs of recovery, though we aren't in the clear yet. We'll continue to keep you posted on our progress.

11:04 AM PST

We're still investigating some potential fixes, but don't have any news to share just yet. Thank you for your continued patience.

10:33 AM PST

We're still on the case, and are doing everything we can to restore all functionality in Slack. We'll be back with more updates soon.

10:03 AM PST

We're continuing to work on getting everything up and running again. Though we saw signs of improvement earlier, unfortunately the fix we had in mind didn't do the trick. We're now looking into another possible fix and will share updates as they become available.

9:33 AM PST

We're still looking into the issues with various Slack services, and will continue to keep you posted on our progress. Apologies again for the disruption to your day.

9:03 AM PST

We have no more news to share just yet, just that we've got all eyes on this. We'll keep you posted with updates as we have them.

8:33 AM PST

Work is still ongoing to get functionality back to normal. We appreciate you sticking with us.

8:03 AM PST

Functionality is almost back to normal, but some users may see some sporadic problems while we work on remediation.

7:33 AM PST

Many of our users may still be having problems with several Slack services. We have all hands on-deck working on a solution. We’ll share more news with you all shortly.

7:03 AM PST

We’re still figuring out the problem some users are having with multiple Slack services. We’re so sorry for the interruption. When we find out more, we’ll let you know.

6:33 AM PST

We’re still investigating the issues with several Slack services, but we don’t have additional information just yet. We’ll be back soon with another update. Sorry for the interruption.

6:03 AM PST

Some customers are still reporting issues with multiple Slack services. We're currently investigating and will keep you posted as soon as we have more information. We're sorry for the disruption.

5:33 AM PST

We received reports of multiple issues regarding Slack's degraded performance, which we are currently investigating.

5:03 AM PST

Some people may be having an issue with Slack. We’re currently investigating and will have more information shortly. Thank you for your patience.

4:54 AM PST