Thursday October 13, 2022

Issue summary:
On October 13, 2022 from 3:00 a.m. to 2:00 p.m. PDT, a number of customers had trouble loading and receiving thread notifications. Some customers may have also had trouble with loading channels, adding users to group direct messages, API hyperlink rendering, and missing Saved items.

A mass de-provisioning event inadvertently increased load across our infrastructure, causing issues with Threads for many users.

As an immediate mitigation strategy, we temporarily deactivated the Threads view for all users.

In parallel, we ended the long running de-provisioning jobs, and significantly scaled up the systems that handle those jobs. We also made several code changes to reduce the effect a de-provisioning job of this scale would have on our infrastructure, and redistributed some of the load across other parts of the system.

The optimisation and redistribution work brought error rates down, which resolved most of the issues with channel loading, group direct messages, hyperlinks, and Saved items that users were experiencing.

At this point, we made the call not to re-enable the Threads view in order to give our recovery efforts more time to restore balance to the system.

Once the load was appropriately balanced across our infrastructure, and we were able to confirm that our up-scaling systems were prepared to handle any future spikes, we re-enabled the Threads view for all users.

7:02 PM PST

We’ve released a fix and all impacted features are up and running once again. If you’re still encountering any trouble, please reload Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux). We appreciate your patience while we sorted this out.

2:40 PM PST

We've identified the cause of this issue and we're currently rolling out a fix for the Threads view. Users should start seeing improvement soon. We'll be back once we have more news to share.

2:03 PM PST

We continue investigating this issue. No additional information to share just yet, but we’ll keep you posted as soon we have an update.

1:34 PM PST

Some users might be able to see the Threads view, but they may experience slowness or trouble loading certain threads within that view. We continue investigating this issue closely and will provide updates as soon as we know more. We apologize for the disruption to your work day.

1:02 PM PST

We're actively working on this issue, but we don't have new information to share at the moment. We appreciate your patience while we get things back to normal.

12:30 PM PST

We're observing significant improvement with notifications, accessing channels/DMs, saved items and API hyperlinks. We're still investigating the issue with loading threads and we'll provide an update as soon as we have news to share. Thank you for your continued patience as we work towards a more ideal solution.

12:02 PM PST

No additional news to share just yet, but we’re focused on getting things back to normal as quickly as we can. We apologize for the continued trouble.

11:31 AM PST

Our investigation is still ongoing at this time. We've discovered that this issue may also be impacting some users' notifications. Thank you for checking in and we'll be back with another update.

10:59 AM PST

We’re still actively investigating this issue, but we don’t have any new information to share at this time. Thanks for sticking with us as we continue to work towards a fix.

10:12 AM PST

We're still actively investigating this issue. We'll provide another update as soon as we have more information to share.

8:53 AM PST

We're still actively investigating this issue. In addition to threads, some users are experiencing issues loading some channels, adding users to group DMs and also seeing their Saved section empty. We'll provide another update as soon as we have more information to share.

8:01 AM PST

We are currently seeing issues with hyperlinks posted by APIs not rendering on the Desktop app. We will continue to investigate this further and we’ll share any additional information as we know more. We appreciate your patience in the meantime.

7:03 AM PST

Some users may be having trouble with loading Slack threads. Additionally, some users may experience their thread badge notifications not clearing correctly. We are sorry for this inconvenience and are actively investigating. We will continue to share additional details as we know more.

6:39 AM PST