On October 18, 2018 at 4 PM PT, we identified a bug in how we were handling accounts being added to IDP groups. This resulted in the deactivation of some user accounts from their workspaces. This also removed their associated integrations, and archived many private channels.
We quickly reverted a code change, and began restoring the affected user accounts, private channels and integrations back to their previous states.
At 5:06 PM PT, we noticed that some servers were overloaded as a result of the restoration efforts we made. This meant that some customers were unable to connect to Slack. We monitored the situation as the server recovered. At 7:41 PM PT, our engineers confirmed functionality was restored for all customers.
On October 19 at 4:32 PM PT, we fully resolved the provisioning issue. Customers who were affected by this issue needed to reload Slack to regain access to their workspaces. We’re continuing to investigate the root cause of these issues to ensure this doesn’t happen in the future.
5:29 PM PST
After some more checks and monitoring we've been given the all-clear. Any affected accounts, integrations, and channel memberships should now be restored. If you’re continuing to see issues please let us know. We know how important your time is and we are truly sorry for taking it today.
11:57 PM PST
All those affected by these SCIM problems should see things restored once more. Though please do let us know if not at email@example.com. We hope to share some more positive news soon.
11:30 PM PST
We are working closely with the affected workspaces to get these SCIM issues rectified. Thanks for holding tight.
10:41 PM PST
The team is still monitoring the changes we've made to fix these SCIM problems. We'll let you know when we have some good news to share.
9:54 PM PST
We are seeing positive signs so far from the fixes to these SCIM issues but we're not out of the woods just yet.
9:19 PM PST
The team continues to work on a fix to restore users, channels, and integrations affected by this issue. We'll keep you updated.
8:40 PM PST
We're working on a fix to restore users, channels, and integrations affected by this issue. Please hold tight.
8:07 PM PST
We really appreciate your patience as we work to get these SCIM issues sorted.
7:34 PM PST
No updates to share just yet with the SCIM issues affecting some workspaces. Though, we'll let you know as soon as we know.
7:02 PM PST
We're still digging through the reports and working with customers to get these SCIM troubles sorted. Please hold tight.
6:31 PM PST
Some workspaces using SCIM may see issues with their Slack accounts. We're working hard to get this rectified quickly.
5:58 PM PST