Wednesday December 14, 2022

Issue summary: From December 13, 2022 at 11:46 AM PST to December 14, 2022 at 7:30 AM PST some customers may have experienced issues while playing videos.

We determined this issue was caused by a recent frontend change that caused a failure to load videos via the streaming protocol HLS (Hypertext Live Streaming), for which no fallback existed.

To resolve this issue, we introduced a fallback to HLS allowing users to play video files with a slightly degraded experience, such as longer load times.

Additionally, we disabled a code optimization technique which fully restored HLS functionality for all users.

10:39 AM PST

We’ve resolved the issue concerning video playback quality affecting some users. You may need to reload Slack (Command/Ctrl + Shift + R) in order for this fix to be reflected on your end. We’re sorry for the disruption we brought to your day.

2:45 PM PST

We are still working to restore full video quality. In the meantime, customers might still notice a degraded experience when playing videos in Slack.

We will be sharing more updates as they become available.

10:02 AM PST

A partial fix was deployed and customers should be able to play videos. If you're still encountering any trouble, please reload Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux).

Some users may still see a degraded experience and we're still working on fully restoring video streaming. Please allow for some extended buffering time and videos will be played. We'll continue our investigation and will be monitoring this closely.

We'll be back with another update in an hour, unless new information becomes available before then. Thank you for your continued patience.

8:49 AM PST

We've identified what's causing the issue for some users when trying to play videos. A temporary fix is being deployed.

Most users should be seeing an improvement soon after reloading Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux).

If you're still encountering any trouble, downloading videos to your device will enable them to be played locally, or you can click the three dots beside the video to open it in a new window. Thank you for your continued patience.

8:06 AM PST

We apologize for the trouble that some customers are experiencing when trying to play videos. We are working hard to fix this issue as soon as possible. In the meantime, downloading videos to your device will enable them to be played locally, or you can click the three dots beside the video to open it in a new window. Thank you for your continued patience.

6:58 AM PST

We're still looking into the cause of the issue that some customers are experiencing when trying to play videos. For now, you can click on the three dots beside the video to open it in a new window or you can download videos directly to your device to play them. Thanks for sticking with us as we continue to work towards a fix.

6:28 AM PST

Our team is working hard to resolve the issue that some customers are experiencing when trying to play videos. In the meantime, downloading videos to your device will enable them to be played locally, or you can click the three dots beside the video to open it in a new window. Sorry for any inconvenience.

5:58 AM PST

We're still actively investigating the issue that some customers are experiencing with playing videos. Apologies for the trouble with this. As a workaround in the meantime, you can either download videos to play them locally on your device, or right-click on the video to open it in a new window. Thank you for your continued patience.

5:28 AM PST

We apologize for the trouble that some customers are experiencing with playing videos. We are working hard to fix this issue as soon as possible. As a workaround, you can download videos to your device and play them locally until this is resolved.

4:53 AM PST

Some customers are having trouble playing videos at the moment, and our team is actively working to resolve this. A workaround is to download videos in order to play them locally. Sorry for any inconvenience.

4:23 AM PST

We are working to resolve the issue with playing videos that some customers are experiencing. In the meantime, downloading videos will enable them to be played locally, until this is resolved. Thanks for bearing with us.

3:53 AM PST

Our team is working hard to resolve the issue that some customers are running into when trying to play videos. Apologies for the trouble with this. As a workaround for now, you can download videos in order to play them locally on your device.

3:23 AM PST

Some customers are hitting issues with playing videos at the moment. Our engineers are working to resolve this issue as quickly as possible. For now, you should be able to download videos, in order to play them. Sorry for the disruption.

2:50 AM PST

Some customers are having trouble playing videos currently. Our team is continuing to investigate this issue. As a workaround in the meantime, you can download videos and play them locally.

2:07 AM PST

Unfortunately some of our customers are currently experiencing problems while playing videos. Our team is investigating and as soon as we have an update we will keep you posted. Apologies for the disruption in the meantime.

1:36 AM PST

We’re receiving reports around trouble playing videos. We’re on the case and will provide another update once we have more details to share. We’re sorry for the inconvenience.

12:35 AM PST

Issue Summary:

On December 14, 2022 from 4:30 AM PST to 9:00 AM PST, we received reports from customers who were unable to receive SMS 2FA codes to log into Slack.

We discovered that the Slack queue on a third party 2FA service, and associated outgoing SMS message errors, were growing at a high rate. Customers who had enabled 2FA via SMS were affected in at least India, China, US, Brazil, and Argentina. We identified that a majority of traffic was coming from bad actors, whose requests led to unusually high volume for our queue.

We prevented these individuals from accessing Slack, while our 2FA service provider implemented a change to further mitigate strain on our queue. These actions resolved the issue for all impacted customers.

Note: This summary has been edited since its initial posting to better align with Slack's communication guidelines.

5:25 PM PST

The issue with two-factor authentication codes via SMS has been resolved. We apologize for any inconvenience.

11:36 AM PST

We're still investigating the issue with users being unable to receive two-factor authentication codes via SMS. We don't have additional information to share just yet. We'll be back shortly with another update.

8:53 AM PST

Some users are currently unable to receive two-factor authentication codes via SMS. We're looking into this with our service provider and hoping to restore functionality again soon.

We apologize for the disruption and we'll share more updates once we have additional information.

7:48 AM PST

Issue summary:
From 10:12 AM PST until 10:25 AM PST on December 14, 2022, some customers had trouble loading Slack workspaces, sending messages, and uploading files.

Prior to this issue, we began removing traffic from an isolated location within a data center region as part of a planned exercise. Upon further investigation, we identified a bug within this process. We then stopped the process, which resolved the issue for all users. Error rates were monitored for a short while prior to announcing the resolution, in order to ensure the issue was fully resolved and would not reoccur.

2:00 PM PST

This issue is resolved, and all users should be able to load Slack, send messages, and upload files once again. We appreciate you bearing with us in the meantime, and we apologize for the disruption to your day.

11:28 AM PST

We've implemented a change and we're seeing signs of recovery. We will continue closely monitoring and sharing news as it becomes available.

10:42 AM PST

Some users may be experiencing trouble loading Slack workspaces, sending messages, and uploading files. We're investigating the situation and we'll let you know once we have more information to share.

10:27 AM PST