On June 28, 2019 at 4:30 a.m. PDT some of our servers became unavailable, causing degraded performance in our job processing system. This resulted in delays or errors with features such notifications, unfurls, and message posting.
During this time, approximately 10-25% of jobs resulted in errors or failure. By 10:00 a.m. PDT, we fully restored message delivery and reduced the error rates to less than 5% as the team continued to work on a full recovery.
At 1:05 p.m. PDT, a separate issue increased server load and dropped a large number of user connections. Reconnection attempts further increased the server load, slowing down customer reconnection. Server capacity was freed up eventually, enabling all customers to reconnect by 1:36 p.m. PDT.
Full service restoration was completed by 7:20 p.m. PDT. During this period, customers faced delays or failure with a number of features including file uploads, notifications, search indexing, link unfurls, and reminders.
Now that service has been restored, the response team is continuing their investigation and working to calculate service interruption time as soon as possible. We’re also working on preventive measures to ensure that this doesn’t happen again in the future. If you’re still running into any issues, please reach out to us at firstname.lastname@example.org.
7:58 PM PST
A fix has been rolled out and all features should be working normally. We are immensely sorry for the disruption to your work day and are continuing the work to make sure this doesn't happen again.
If you are still seeing any issues, please don't hesitate to drop us a line via our help page or email@example.com — we're here to help.
8:15 PM PST
We're seeing positive signs with some changes being rolled out, and are continuing to work to get things back to 100%. We really appreciate your patience.
7:27 PM PST
Customers may see degraded functionality for several other features, including notifications, reminders and workspace member lists — we're working to restore that. File uploads should now work after a refresh (Cmd/Ctrl + R).
6:14 PM PST
We’re still working hard to restore file upload functionality. Thank you for your continued patience.
4:54 PM PST
File uploads are still not working for some users but refreshing Slack (Cmd/Ctrl + R) may restore functionality.
3:56 PM PST
We don't have any updates right now. Refreshing Slack (Cmd/Ctrl + R) should restore connectivity but we’re still working on file uploads.
3:25 PM PST
We expect users to be able to connect to Slack after a refresh (Cmd/Ctrl + R) but we are still experiencing some issues with file uploads.
2:54 PM PST
We are still seeing reports from many users unable to connect to Slack. We truly appreciate your patience while we monitor the situation.
2:24 PM PST
Some users have reported problems loading Slack. We'll continue to update you on our progress.
1:54 PM PST
Some users have reported intermittent connectivity issues. We remain hard at work to come to a full resolution.
1:24 PM PST
Still no news to share on a resolution, but please continue to let us know if you're running into any trouble after reloading. We'll return with more updates soon.
12:51 PM PST
We're still working on getting things completely fixed up, but for now reloading should help. If this doesn't lead to improvements, though, drop us a line at firstname.lastname@example.org.
12:21 PM PST
We’re seeing that functionality may be restored after reloading (Cmd/Ctrl + R). If you're still running into trouble after giving that a try, please let us know at email@example.com.
11:51 AM PST
No news to share just yet, but we remain hard at work to get everything running smoothly again. We appreciate your patience in the meantime.
11:34 AM PST
We're seeing some signs of recovery, though we aren't in the clear yet. We'll continue to keep you posted on our progress.
11:04 AM PST
We're still investigating some potential fixes, but don't have any news to share just yet. Thank you for your continued patience.
10:33 AM PST
We're still on the case, and are doing everything we can to restore all functionality in Slack. We'll be back with more updates soon.
10:03 AM PST
We're continuing to work on getting everything up and running again. Though we saw signs of improvement earlier, unfortunately the fix we had in mind didn't do the trick. We're now looking into another possible fix and will share updates as they become available.
9:33 AM PST
We're still looking into the issues with various Slack services, and will continue to keep you posted on our progress. Apologies again for the disruption to your day.
9:03 AM PST
We have no more news to share just yet, just that we've got all eyes on this. We'll keep you posted with updates as we have them.
8:33 AM PST
Work is still ongoing to get functionality back to normal. We appreciate you sticking with us.
8:03 AM PST
Functionality is almost back to normal, but some users may see some sporadic problems while we work on remediation.
7:33 AM PST
Many of our users may still be having problems with several Slack services. We have all hands on-deck working on a solution. We’ll share more news with you all shortly.
7:03 AM PST
We’re still figuring out the problem some users are having with multiple Slack services. We’re so sorry for the interruption. When we find out more, we’ll let you know.
6:33 AM PST
We’re still investigating the issues with several Slack services, but we don’t have additional information just yet. We’ll be back soon with another update. Sorry for the interruption.
6:03 AM PST
Some customers are still reporting issues with multiple Slack services. We're currently investigating and will keep you posted as soon as we have more information. We're sorry for the disruption.
5:33 AM PST
We received reports of multiple issues regarding Slack's degraded performance, which we are currently investigating.
5:03 AM PST
Some people may be having an issue with Slack. We’re currently investigating and will have more information shortly. Thank you for your patience.
4:54 AM PST