Thursday February 20, 2020
Incident
Degraded performance and messages failing to send intermittently
Issue Summary:
From February 19, 2020 at 2:08 a.m. PST until February 20 at 12:45 p.m. PST, some customers experienced trouble sending messages. After investigating further, we found that a newly added API included an expensive query, causing other queries to fail and build up in a backlog. We ultimately removed this API from our system, which resolved the issue for all impacted customers.
10:17 AM PST
We've rolled out a fix, and customers should no longer be having trouble sending messages or experiencing generalized performance issues. We're very sorry for any interruption to your day, and if you are still experiencing any trouble, please let us know at feedback@slack.com
12:56 PM PST
We're nearing a fix for these issues, however we aren't quite out of the woods just yet. We'll let you know as soon as we have more news to share.
12:27 PM PST
The investigation around messages failing to send is ongoing, but we are seeing improvements as we move forward. We appreciate your patience as we continue our efforts to get this up and running again.
8:39 AM PST
We continue to notice spikes in messages failing to send. The issue might also cause degraded performance across Slack. We're sorry for the trouble and working to resolve this.
6:01 AM PST
We're still working to understand the cause of the problem and to mitigate the occasional spikes in message failures.
5:22 AM PST
We continue to investigate the source of the higher rate of messages failing to send.
4:31 AM PST
We're still looking into what's been causing the spikes in failed messages. We appreciate the patience and are sorry for the disruption.
4:01 AM PST
We're observing occasional spikes in messages failing to send. Our engineers are investigating the problem. Apologies for the trouble.
3:30 AM PST
Services affected
Connectivity
Messaging
Status
Incident