Tuesday March 9, 2021

Issue summary:
On March 9, 2021 from 5:30 a.m. to 9:03 a.m. PST, some customers experienced delays, such as uploading files and receiving notifications.

We identified a server upgrade that resulted in tasks taking longer than they should. This upgrade was reverted, and the issue was resolved.

Note: We've updated the title for this issue.

2:28 PM PST

We've confirmed the fix was successful, and affected customers should no longer be experiencing delays. Thank you for your patience and understanding while we got to the bottom of things.

9:13 AM PST

We believe we've identified the source of the issue and deployed a fix. Affected customers should start seeing improved performance, and we'll continue to keep a close eye on the situation. We'll provide another update shortly.

8:47 AM PST

We're still investigating the issue around processing delays. We don't have additional information just yet, but we'll be back in 30 minutes with another update. Thank you for bearing with us.

8:17 AM PST

We're experiencing an issue that's resulting in delays, such as notifications, as well as messages from apps and integrations, taking longer to be delivered.

Note: We've updated this message to clarify what the issue resulted in as well as note the issue wasn't isolated to the two examples provided.

7:47 AM PST

Issue summary:
On March 9, 2021 from around 5:39 a.m. to 6:27 a.m. PST, some customers experienced issues using Workflow Builder. These issues included seeing an error when using a workflow started by a channel action and not seeing recent changes when modifying a workflow. Some workflow failures succeeded on a retry, while others failed altogether.

These issues were the result of congestion in some areas of the Workflow Builder infrastructure, which also caused the trouble with this feature on March 3, 2021.

As a temporary remediation, we removed this congestion from the affected areas of the infrastructure to allow Workflow Builder to start working as expected. We’re continuing to work to resolve the root cause to prevent this congestion from happening again. We’ve made two changes so far, and we’ve seen a reduction in the amount of congestion due to these changes.

Note: We’ve updated this summary to include specific examples of what the trouble looked like in Slack as well as clarify our remediation steps. We’ve also updated the title.

6:51 PM PST

Customers should no longer be noticing issues when using Workflow Builder. If it’s still not quite right, please try a refresh using CMD/CTRL + R.

Thanks again for your patience and understanding while we fixed this issue. We'll be posting a summary with more details as soon as possible.

6:38 AM PST

Customers may be experiencing issues using Workflow Builder within Slack. We apologize for the disruption. We're investigating and working to resolve this as fast as we can.

6:08 AM PST