Friday October 16, 2020

Incident

Degraded performance and connection issues

Issue summary:
From October 16, 2020 from 12:10am to 1:01pm PDT, some users encountered trouble when using Slack. Problems included degraded performance, such as slowness when switching channels, seeing a "Something's gone awry" error message on desktop, and being unexpectedly signed out of the mobile app.

We discovered there was an issue with reading and writing to our cache tier, which then put extra load on our database tier. This extra load on our database tier resulted in some requests being slow or failing altogether.

We first mitigated the issue by adding additional database capacity to reduce the load on our database tier. We then resolved the root issue with reading and writing to our cache tier to bring database load back to normal levels.

If you would like to receive a full Root Cause Analysis (RCA) report, please reach out to feedback@slack.com to request one.

1:27 PM PST

We've confirmed customers should no longer be encountering trouble. We're working to fully understand the core issue, and we'll follow-up here with a summary once we have more information.

We sincerely apologize for any disruption, and appreciate your understanding and patience.

11:52 AM PST

We're continuing to work towards a full resolution on our end. At this time, customers should still be seeing little to no impact. We believe things should remain operational and we'll check back in an hour or sooner if anything changes.

10:37 AM PST

We've made some improvements on our end, and customers should be experiencing little to no impact now. The issue is not resolved, but our team is working to bring things into a controlled state while investigating.

10:05 AM PST

We're continuing to investigate the degraded performance. Although we're seeing some error rates improve, we're not at a resolution just yet.
We apologize for this inconvenience and will continue to post updates every 30 minutes.

9:38 AM PST

Users may once again have trouble connecting to Slack and are experiencing significantly degraded performance. Our team is still on the case and we will keep you posted as soon as we have more information to share.

9:14 AM PST

We are still investigating and apologize that we don't have a firm update to share as of yet. Please know that all hands are on deck and we're doing all that we can to get things up and running.

8:33 AM PST

Most users should now be able to connect but may still run into issues with degraded performance. Thanks for your patience as we continue to look into it.

8:03 AM PST

We’re continuing to investigate the degraded performance. Users may have issues sending messages, or experience API failures or general slowness. We are still investigating, and we’ll let you know where we’re at in 30 minutes.

7:39 AM PST

We're still investigating the cause of the performance issues that users may be facing. We appreciate you bearing with us and we'll keep you updated as we have more news.

7:03 AM PST

We don't have any new information to share at the moment. We'll be back with an update as soon as we do. Thank you for your patience.

6:33 AM PST

No additional news to share at this stage, but we’re focused on getting things back to normal as quickly as possible. We’ll be back with another update in 30 minutes.

6:04 AM PST

Some users are struggling to connect to Slack, or facing trouble with the app operating slowly. We are currently investigating, and apologise for the inconvenience caused.

5:33 AM PST

Services affected

Calls

Messaging

Search

Link Previews

Apps/Integrations/APIs

Posts/Files

Workspace/Org Administration

Login/SSO

Notifications

Connections

Status

Incident