From October 16, 2020 from 12:10am to 1:01pm PDT, some users encountered trouble when using Slack. Problems included degraded performance, such as slowness when switching channels, seeing a "Something's gone awry" error message on desktop, and being unexpectedly signed out of the mobile app.
We discovered there was an issue with reading and writing to our cache tier, which then put extra load on our database tier. This extra load on our database tier resulted in some requests being slow or failing altogether.
We first mitigated the issue by adding additional database capacity to reduce the load on our database tier. We then resolved the root issue with reading and writing to our cache tier to bring database load back to normal levels.
If you would like to receive a full Root Cause Analysis (RCA) report, please reach out to firstname.lastname@example.org to request one.
1:27 PM PST
We've confirmed customers should no longer be encountering trouble. We're working to fully understand the core issue, and we'll follow-up here with a summary once we have more information.
We sincerely apologize for any disruption, and appreciate your understanding and patience.
11:52 AM PST
We're continuing to work towards a full resolution on our end. At this time, customers should still be seeing little to no impact. We believe things should remain operational and we'll check back in an hour or sooner if anything changes.
10:37 AM PST
We've made some improvements on our end, and customers should be experiencing little to no impact now. The issue is not resolved, but our team is working to bring things into a controlled state while investigating.
10:05 AM PST
We're continuing to investigate the degraded performance. Although we're seeing some error rates improve, we're not at a resolution just yet.
We apologize for this inconvenience and will continue to post updates every 30 minutes.
9:38 AM PST
Users may once again have trouble connecting to Slack and are experiencing significantly degraded performance. Our team is still on the case and we will keep you posted as soon as we have more information to share.
9:14 AM PST
We are still investigating and apologize that we don't have a firm update to share as of yet. Please know that all hands are on deck and we're doing all that we can to get things up and running.
8:33 AM PST
Most users should now be able to connect but may still run into issues with degraded performance. Thanks for your patience as we continue to look into it.
8:03 AM PST
We’re continuing to investigate the degraded performance. Users may have issues sending messages, or experience API failures or general slowness. We are still investigating, and we’ll let you know where we’re at in 30 minutes.
7:39 AM PST
We're still investigating the cause of the performance issues that users may be facing. We appreciate you bearing with us and we'll keep you updated as we have more news.
7:03 AM PST
We don't have any new information to share at the moment. We'll be back with an update as soon as we do. Thank you for your patience.
6:33 AM PST
No additional news to share at this stage, but we’re focused on getting things back to normal as quickly as possible. We’ll be back with another update in 30 minutes.
6:04 AM PST
Some users are struggling to connect to Slack, or facing trouble with the app operating slowly. We are currently investigating, and apologise for the inconvenience caused.
5:33 AM PST