Wednesday August 2, 2023

Issue summary:
On August 2, 2023 from 9:01 AM PDT to 11:00 AM PDT, customers may have experienced issues with blurry images in Slack, as well as trouble uploading files. A small percentage of customers may have also experienced errors or slowness when loading channels, sending messages, accessing admin pages, and connecting to Slack.

An issue with a routine database cluster migration inadvertently caused a decrease in capacity. Unfortunately, this coincided with a scheduled job to process a large number of custom status requests, an activity that commonly occurs at the top of each hour. The number of incoming requests exceeded the unintentionally reduced capacity of the database cluster, leading to errors and sluggishness for some actions in Slack. As these requests piled up, other Slack services began to experience issues as well, including a small number of users being unable to connect.

We paused the scheduled custom status job to immediately mitigate the load on the affected database cluster. This resolved some of the errors affecting Slack services. Once we established we had sufficient capacity to handle the scheduled job, we resumed processing the custom status requests, fully restoring normal service for all affected users.

Once the load was balanced and custom status processing had returned to normal, we deployed a fix to correct the issue with the database cluster migration, bringing capacity back to the expected level.

7:32 PM PST

We’ve identified the root cause and rolled out a fix. Users should no longer be experiencing any trouble connecting to Slack.

Thank you very much for your patience while we worked on getting this fix out the door.

4:47 PM PST

Based on all metrics we're monitoring, there is no remaining impact to users at this time, and Slack should be operating as expected.

With this said, we are actively working on reversing mitigative changes implemented during our investigation to bring our systems to their previous state, and we're continuing to monitor to ensure issues do not reoccur.

We will be back with an update when we have more information to share. Thank you for sticking with us.

11:32 AM PST

We’re continuing to see signs of improvement and are still monitoring the situation.

We’ll confirm once the issue is fully resolved or we have another update to share.

Thank you for your continued patience.

11:14 AM PST

We have taken steps to mitigate impact to users and are seeing signs of improvement.

We will be back once we have another update to share.

We apologize for the disruption and appreciate you bearing with us in the meantime.

10:44 AM PST

We’re still actively investigating this issue, but we don’t have any new information to share at this time.

Thanks for sticking with us as we continue to work towards a fix.

10:14 AM PST

There are systemwide issues across Slack that may result in users being unable to connect to Slack.

Users may be experiencing issues loading threads, admin pages, images, running workflows and sending messages.

We're still investigating, and we'll share updates as we have more information. We're sorry for the disruption this may cause.

9:42 AM PST

Users may be experiencing issues loading threads, admin pages and images. There may also be issues sending messages and running workflows.

We're currently investigating, and we'll share updates as we have more information. We're sorry for the disruption.

9:21 AM PST