Tuesday February 22, 2022

Issue summary:
From 5:52 PM PST on February 22, 2022 until 4:41 PM PST on February 23, 2022, some customers had trouble receiving their reminders, and Google Calendar for Teams was not posting in-channel reminders.

We traced the problem to a change we'd implemented. We reverted this, resolving the issue for all affected customers.

9:31 PM PST

We've resolved the issue, and all impacted customers should now be able to receive reminders. You may need to reload Slack (Cmd/Ctrl + Shift + R) to see the fix on your end. Thanks for bearing with us and we apologize for keeping you away from your work.

8:53 AM PST

We are seeing improvements and newly set reminders should be working as normal again. We are still working to restore full functionality to older reminders for affected customers. We'll be back with another update as soon as we have more information.

5:11 PM PST

Less than 1% of users are having trouble receiving reminders. We are continuing to look into this and will provide an update as soon as there’s news to share.

9:44 AM PST

We're still investigating the reports about using reminders. We do apologize for any disruption.

7:50 AM PST

We’re receiving reports around trouble using reminders. We’re on the case and will provide another update once we have more details to share. We’re sorry for the inconvenience.

11:07 PM PST

Issue summary:
On February 22, 2022 from 6:00 AM PST to 9:14 AM PST, customers may not have been able to access Slack.

A configuration change inadvertently lead to a sudden increase in activity on our database infrastructure. Due to this increased activity, the affected databases failed to serve incoming requests to connect to Slack. We introduced tighter rate limits on connection requests to reduce the load on the system. This meant that some people could not access Slack at all, but also that Slack would continue working for those who were already connected.

Once the system had stabilized, we began lifting these rate limits to enable more connections to Slack. However, we moved too quickly and the increased activity affected the system again. We reinstated the rate limits and redirected some traffic to the database replicas to relieve the demand on our primary databases.

With traffic distributed across the additional databases, we gradually lifted the rate limits to allow more people to connect to Slack. This allowed the system to recover while steadily restoring full access to all users. By 9:14 AM PST, everyone should have been able to connect to Slack again after reloading their app or browser tab.

We continued to closely monitor the situation until we had full confidence the issue was resolved and no further impact would reoccur.

We know our customers rely on Slack. If you'd like to receive a full Root Cause Analysis (RCA) report, please reach out to feedback@slack.com to request one.

11:18 PM PST

We've resolved the issue, and all impacted customers should now be able to access Slack. You may need to reload Slack (Cmd/Ctrl + Shift + R) to see the fix on your end. If that doesn't work, try clearing cache (Help > Troubleshooting > Clear Cache and Restart from the app menu). Thanks for bearing with us and we apologize for the disruption to your work day!

11:24 AM PST

We're continuing to see improvements, but we still have some work to do. If you're still encountering any trouble, please reload Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux) or clear the app cache (Help > Troubleshooting > Clear Cache and Restart on desktop, Preferences > Advanced > Reset Cache on mobile).

10:44 AM PST

We're continuing to see signs of improvement and are monitoring the situation. If you're still encountering any trouble, please reload Slack or clear the app cache.

10:08 AM PST

We're seeing signs of improvement. Please try reloading Slack, and if not a cache reset. We’re still monitoring the situation. We’ll confirm once this issue is fully resolved.

9:07 AM PST

Some customers are unable to load Slack. We’re still actively investigating this issue, but we don’t have any new information to share at this time. We’ll keep you posted as soon as we have an update.

8:24 AM PST

We’re still actively investigating this issue, but we don’t have any new information to share at this time. We’ll be back with another update in 30 minutes.

7:54 AM PST

Slack is not loading for some users. We are continuing to investigate the cause and will provide more information as soon as it's available.

7:23 AM PST

We're still working towards a full resolution. We'll be back with another update soon. Thank you for your patience.

6:44 AM PST

We’re investigating the issue where Slack is not loading for some users. We’re looking into the cause and will provide more information as soon as it's available.

6:25 AM PST