Tuesday April 19, 2022

Issue summary:
From April 19, 2022 at 4:38 PM PDT to April 20, 2022 at 12:36 PM PDT, our SCIM endpoint began deactivating some users.

This was the result of a code change in a recent patch to fix a bug affecting the way SCIM handled user profile updates.

We reverted this code change to prevent new deactivations. We then restored affected user accounts, group memberships, and integrations to their prior states.

Note: Since initially posting, we've edited the times included in this summary to accurately reflect the duration of the issue.

Note: We discovered a small percentage of users who were still deactivated until April 21, 2022 at 10:40 PM PDT, and we have restored their accounts.

4:48 PM PST

The SCIM issue has been resolved and all affected users have been reactivated. We apologize for the disruption to your day.

12:59 PM PST

We are continuing to monitor the situation and verify that all affected users are reactivated. We're working on restoring group membership and impacted integrations. We'll be back once we have another update to share.

9:48 AM PST

The team is still ensuring that all affected users are reactivated. While we are continuing to work on restoring group membership and impacted integrations, we appreciate your patience.

8:46 AM PST

We are verifying that all affected users are now reactivated. Our team will now begin work on restoring group membership and impacted integrations. We appreciate your patience while we work to get this resolved.

7:43 AM PST

We're still doing our best to restore all affected users and will share new updates here as soon as we have them.

We appreciate your continued patience while we work to get this resolved.

4:44 AM PST

We appreciate your patience as our team continues to work to restore all affected users.

We will keep you posted as soon as we have further updates to share.

11:59 PM PST

Our team is currently working on restoring all affected users.

We’ll be back once we have another update to share. Thank you for your patience as we continue to work towards a fix.

11:47 PM PST

Our team have now identified all affected users and workspaces. We're still monitoring the situation, and we'll report back with more information in 30 minutes. Thank you for holding tight while we work on this.

10:34 PM PST

We've identified the root cause of this issue, and SCIM events should now be working as expected. We are actively investigating remediation for affected customers.

Thank you for your patience while we look into this. We'll share another update in 30 minutes.

9:51 PM PST

We’re investigating an issue with SCIM deactivating users, and SCIM updates not working as expected. We’re looking into the cause and will provide more information as soon as it's available.

8:46 PM PST

Issue summary:
On April 19, 2022 from approximately 1:00 PM PDT to 3:50 PM PDT, some Enterprise customers may have experienced delays related to reminders, notifications and other task-related features.

These issues were caused when the Primary Owner’s email address of a large load-test org was updated, which resulted in the delayed processing of Enterprise job queues. After identifying the issue, we released a fix to resolve job queue delays and return Enterprise job processing to normal.

Note: timestamps in this summary were edited after posting to reflect the accurate duration of this issue.

7:48 PM PST

We've traced the performance issues back to a job that had backed up the queue responsible for handling actions taken by users and administrators in Enterprise Grid organizations.

The backlog of tasks have completed and all features should be working as expected, though we recommend reloading the Slack app (Cmd/Ctrl + Shift + R) just in case. Thank you for your patience.

4:53 PM PST

Enterprise Grid customers may be experiencing trouble using certain features in Slack. This may include setting custom statuses and creating channels, and we’ll continue to provide updates as we learn more.

2:57 PM PST