Tuesday March 23, 2021
Incident
Some users unable to connect or experiencing degraded service
Issue summary:
From March 22, 2021 around 1:26 p.m. to 4:25 a.m PDT on March 23, 2021, a small number of users encountered issues in Slack, such as slowness and trouble connecting.
This trouble began after we introduced a change intended to test performance improvements to our network layer. We reverted the change, then began staged restarts of a service on the servers. We continued to monitor the situation while we moved to a full remediation.
We acknowledge the delay in posting this summary, and we appreciate your patience.
Note: We've updated the title and services affected to reflect that the trouble wasn't only with connecting to Slack.
4:17 PM PST
We’ve resolved the connectivity issue and users should be able to access Slack as normal now. Apologies for the disruption we caused, and thank you for your patience.
5:48 AM PST
Some users may be experiencing difficulties with connecting to their workspaces. The situation is already improved, but we’re keeping a close eye on things.
4:43 AM PST
Services affected
Login/SSO
Connectivity
Messaging
Link Previews
Files
Notifications
Huddles
Search
Apps/Integrations/APIs
Workspace/Org Administration
Status
Incident
Incident
Trouble loading messages after converting group direct messages to private channels
Issue summary:
On March 23, 2021 starting around 5:15 p.m. PDT, a small number of customers experienced issues loading messages after converting group direct messages to private channels.
We determined that a code change to our database infrastructure inadvertently caused this issue. We reverted the code change around 1:43 a.m. PDT on March 24, 2021, which prevented further occurrences of the issue. We also ran remediation scripts so that the impacted channels could load messages once again.
This issue was resolved by or before 7:20 p.m. PDT on March 24, 2021 for most users (a reload might've been needed to see the messages). However, we discovered that some users were still impacted, so we worked to resolve the outstanding trouble. By March 26, 2021 around 3:00 p.m. PDT, the issue was resolved for all affected users.
Note: We've updated this summary to provide a more accurate timeline of events. In our message below, we said the issue was resolved on March 24, 2021, but we discovered this wasn't the case for all affected users. This summary has been updated to reflect our findings.
11:12 PM PST
We've resolved the issue, and messages should now load normally after converting group direct messages to private channels. If you're still affected, please reload Slack with Command/Control + Shift + R. Thank you for bearing with us.
8:30 PM PST
We've addressed the root cause, so converting group direct messages to private channels should work as expected moving forward.
We're working to resolve the trouble for those converted group direct messages already impacted, and we'll be back once this work is complete. We apologize for any disruption.
Note: We've updated this message to clarify we're still working on resolving the trouble for some users, and we'll be back once it's resolved. We've also slightly revised the title. Feel free to contact us at feedback@slack.com if there's anything we can assist with.
2:36 AM PST
Some customers may be experiencing issues with loading messages after converting group direct messages to private channels. We’re investigating and will be back with an update shortly.
11:31 PM PST
Services affected
Messaging
Status
Incident