Thursday February 20, 2020

Incident

Degraded performance and messages failing to send intermittently

Issue Summary:

From February 19, 2020 at 2:08 a.m. PST until February 20 at 12:45 p.m. PST, some customers experienced trouble sending messages. After investigating further, we found that a newly added API included an expensive query, causing other queries to fail and build up in a backlog. We ultimately removed this API from our system, which resolved the issue for all impacted customers.

10:17 AM PST

We've rolled out a fix, and customers should no longer be having trouble sending messages or experiencing generalized performance issues. We're very sorry for any interruption to your day, and if you are still experiencing any trouble, please let us know at feedback@slack.com

12:56 PM PST

We're nearing a fix for these issues, however we aren't quite out of the woods just yet. We'll let you know as soon as we have more news to share.

12:27 PM PST

The investigation around messages failing to send is ongoing, but we are seeing improvements as we move forward. We appreciate your patience as we continue our efforts to get this up and running again.

8:39 AM PST

We continue to notice spikes in messages failing to send. The issue might also cause degraded performance across Slack. We're sorry for the trouble and working to resolve this.

6:01 AM PST

We're still working to understand the cause of the problem and to mitigate the occasional spikes in message failures.

5:22 AM PST

We continue to investigate the source of the higher rate of messages failing to send.

4:31 AM PST

We're still looking into what's been causing the spikes in failed messages. We appreciate the patience and are sorry for the disruption.

4:01 AM PST

We're observing occasional spikes in messages failing to send. Our engineers are investigating the problem. Apologies for the trouble.

3:30 AM PST

Services affected

Connections

Messaging

Status

Incident