Monday January 4, 2021

Issue summary:
On January 4, 2021 from around 10:00 a.m. to 12:30 p.m. PST, Google Calendar, Outlook Calendar, email notifications, email forwarding and replies, and creating new email addresses for channels were fully unavailable.

While working to resolve issues involved with the outage on January 4, 2021, we determined the best course of action was to temporarily disable these features. We wanted to work on remediations while minimizing risk to all systems involved.

We worked to restore service to these features and reached a full resolution by 7:16 p.m. PST, but some features became available earlier on.

Note: We've updated this summary and title, as we've gained a better understanding of what features were impacted and when.

6:55 PM PST

Both Outlook Calendar and Google Calendar should now be working as expected.

We're terribly sorry for the disruption, and we very much appreciate your patience. We'll follow up with a summary as soon as we can.

9:05 PM PST

You should now be able to reply to notification emails delivered after 4:45 p.m. PST.

We're still working to bring the calendar apps fully back online. We'll return with more information as soon as we have news to share.

Thank you for bearing with us.

5:18 PM PST

Setting up a new email forwarding rule right now may also be affected by the email notifications issue. Our team is continuing to investigate this.

We're close to resolving the issue with the calendar apps, and we'll provide an update as soon as we have one.

4:00 PM PST

Users who've joined a workspace and been invited to a workspace should now be receiving email notifications successfully. However, they won't be able to reply to the emails.

The issues with Google Calendar and Outlook Calendar are still being worked on. We'll be back once we have some more information.

2:50 PM PST

We're closer to getting Google Calendar working as expected, but we're not at a resolution yet. We're also continuing to work to resolve the trouble with Outlook Calendar and email notifications.

1:47 PM PST

For users who've joined a workspace, email notifications should be delivered. However, these users won't be able to reply to the emails.

For users who've been invited to a workspace, but haven't joined, email notifications won't be sent at this time.

We'll be back once we have more information about the trouble with the calendars and email notifications.

12:46 PM PST

Google Calendar and Outlook Calendar aren't working as expected, and some email notifications (e.g. a notification of a new direct message in Slack) are failing to send. We're working to resolve this issue, and we're sorry for the trouble.

12:12 PM PST

Issue summary:
Starting around 6:00 a.m. PST on January 4, 2021, some customers started experiencing occasional errors and increased latency while using Slack. Around 7:00 a.m. PST there was a rapid increase in errors and Slack wasn’t usable for all customers.

Around 8:13 a.m. PST, we addressed an issue with our provisioning service and began provisioning healthy servers once again to address traffic requests. From there, at 8:45 a.m. PST, some customers began to see improvements, but others who were trying to launch their Slack clients were unable to do so. By around 9:15 a.m. PST most customers were able to use Slack again. We continued to experience elevated errors until 10:40 a.m. PST, after which all customers were able to use Slack again.

We also discovered some customers were stuck on a webpage in the Slack desktop app. This is a separate bug that’s being investigated, but was heightened during the outage. Troubleshooting steps, such as restarting, forcing quitting Slack from Activity Monitor or Task Manager, or clearing cache, allowed affected customers to access the app once again.

We know our customers rely on Slack. Our top priorities are working alongside our cloud vendors to fully understand the underlying causes of the outage and putting safeguards in place to prevent similar problems from happening again. If you’d like to receive a full Root Cause Analysis (RCA) report, please reach out to feedback@slack.com to request one.

5:04 PM PST

For any customers unable to access Slack, please try refreshing by using CMD/CTRL + SHIFT + R. It may also help to try closing Slack via the Task Manager or Activity Monitor. If you encounter any trouble, please let us know at feedback@slack.com.

We'll have a separate post on this site for the outstanding calendar apps and email notifications trouble.

We're truly sorry for the disruption today. We'll be back with a summary of this issue.

Note: We've edited this message to ask users to refresh their clients entirely, as opposed to the reload instructions we shared before (CMD/CTRL + R). It's also been edited to ask users to try force closing their app.

12:10 PM PST

We're seeing improvements with error rates on our side, and we believe affected customers should be able to access Slack.

We're continuing to work to resolve the trouble with calendar apps and email notifications.

11:28 AM PST

For those customers who are unable to access Slack (e.g. seeing a blank screen or error), please try reloading (CTRL/CMD + R). If you can access Slack, please hold off on reloading at this time.

In addition, Google Calendar and Outlook Calendar aren't working as expected, and some email notifications (e.g. a notification of a new direct message in Slack) are failing to send.

Note: We've updated this message to specify holding off on reloading if you can access Slack.

10:51 AM PST

Some customers may be able to connect, but may also experience degraded performance. We're continuing to work to resolve the issue.

10:00 AM PST

While the issue is largely still ongoing, we believe some customers may see improvement in connecting to Slack after a refresh (CTRL/CMD + R).

9:29 AM PST

There are no changes to report as of yet. We're still all hands on deck and continuing to dig in on our side. We'll continue to share updates every 30 minutes until the incident has been downgraded.

8:52 AM PST

We're continuing to investigate connection issues for customers, and have upgraded the incident on our side to reflect an outage in service. All hands are on deck on our end to further investigate. We'll be back in a half hour to keep you posted.

8:20 AM PST

We’re still investigating the ongoing connectivity issues with Slack. There's no additional information to share just yet, but we’ll follow up in 30 minutes. Thanks for bearing with us.

7:44 AM PST

Customers may have trouble loading channels or connecting to Slack at this time. Our team is investigating and we'll follow up with more information as soon as we have it. We apologize for any disruption caused.

7:14 AM PST