Thursday October 18, 2018
Incident
Users experiencing slowness in the app
Technical Summary
On October 18, 2018 at 4 PM PT, we identified a bug in how we were handling accounts being added to IDP groups. This resulted in the deactivation of some user accounts from their workspaces. This also removed their associated integrations, and archived many private channels.
We quickly reverted a code change, and began restoring the affected user accounts, private channels and integrations back to their previous states.
At 5:06 PM PT, we noticed that some servers were overloaded as a result of the restoration efforts we made. This meant that some customers were unable to connect to Slack. We monitored the situation as the server recovered. At 7:41 PM PT, our engineers confirmed functionality was restored for all customers.
On October 19 at 4:32 PM PT, we fully resolved the provisioning issue. Customers who were affected by this issue needed to reload Slack to regain access to their workspaces. We’re continuing to investigate the root cause of these issues to ensure this doesn’t happen in the future.
5:33 PM PST
We've been given the all-clear. Everyone should be able to use Slack at full speed once more. If you are still experiencing slow performance of the app please try an app reload (ctrl or cmd + R) and let us know if this doesn't help. We are truly sorry if your day was affected by this issue. We will do all we can to make sure this doesn't happen again.
11:40 PM PST
No new news to share just yet, but we're continuing our efforts. We truly appreciate your patience.
10:55 PM PST
No further developments to share but we wanted to let you know we're doing all we can to get this app slowness fixed as soon as possible. Thanks again.
10:17 PM PST
Thank you for sticking with us through this. We hope to have the app running at regular speeds soon.
9:29 PM PST
Nothing further to add, sadly. The team continues to monitor things and work on a solution for this app slowness.
8:44 PM PST
To those affected by this app slowness we're truly sorry for the disruption. We're working hard to get this rectified.
8:12 PM PST
Thank you again for your patience. We're doing all we can to get things working at normal speeds once more. We're working hard to get this rectified.
7:30 PM PST
We're still investigating the cause of this general slowness with the app and hope to have some good news soon.
6:58 PM PST
Thank you for your patience here. Fixing this slowness in the app is the highest priority for the team right now.
6:28 PM PST
Nothing new to share just yet. The team are monitoring things and we'll let you know as soon as we get more information.
5:55 PM PST
Some people have reported general slowness in their workspaces. Apologies for this hassle. We'll get this fixed as soon as possible.
5:24 PM PST
Services affected
Connectivity
Messaging
Files
Search
Status
Incident
Incident
SCIM issues affecting some workspaces
Technical Summary
On October 18, 2018 at 4 PM PT, we identified a bug in how we were handling accounts being added to IDP groups. This resulted in the deactivation of some user accounts from their workspaces. This also removed their associated integrations, and archived many private channels.
We quickly reverted a code change, and began restoring the affected user accounts, private channels and integrations back to their previous states.
At 5:06 PM PT, we noticed that some servers were overloaded as a result of the restoration efforts we made. This meant that some customers were unable to connect to Slack. We monitored the situation as the server recovered. At 7:41 PM PT, our engineers confirmed functionality was restored for all customers.
On October 19 at 4:32 PM PT, we fully resolved the provisioning issue. Customers who were affected by this issue needed to reload Slack to regain access to their workspaces. We’re continuing to investigate the root cause of these issues to ensure this doesn’t happen in the future.
5:29 PM PST
After some more checks and monitoring we've been given the all-clear. Any affected accounts, integrations, and channel memberships should now be restored. If you’re continuing to see issues please let us know. We know how important your time is and we are truly sorry for taking it today.
11:57 PM PST
All those affected by these SCIM problems should see things restored once more. Though please do let us know if not at feedback@slack.com. We hope to share some more positive news soon.
11:30 PM PST
We are working closely with the affected workspaces to get these SCIM issues rectified. Thanks for holding tight.
10:41 PM PST
The team is still monitoring the changes we've made to fix these SCIM problems. We'll let you know when we have some good news to share.
9:54 PM PST
We are seeing positive signs so far from the fixes to these SCIM issues but we're not out of the woods just yet.
9:19 PM PST
The team continues to work on a fix to restore users, channels, and integrations affected by this issue. We'll keep you updated.
8:40 PM PST
We're working on a fix to restore users, channels, and integrations affected by this issue. Please hold tight.
8:07 PM PST
We really appreciate your patience as we work to get these SCIM issues sorted.
7:34 PM PST
No updates to share just yet with the SCIM issues affecting some workspaces. Though, we'll let you know as soon as we know.
7:02 PM PST
We're still digging through the reports and working with customers to get these SCIM troubles sorted. Please hold tight.
6:31 PM PST
Some workspaces using SCIM may see issues with their Slack accounts. We're working hard to get this rectified quickly.
5:58 PM PST
Services affected
Login/SSO
Workspace/Org Administration
Status
Incident
Incident
Trouble with Events API, notifications, unfurls, and threads
Technical Summary:
Starting on October 17, 2018 at 11:28 AM PT, some customers were seeing delays with certain features, including link unfurls, notifications, and integrations. After noticing a large increase in backlogged jobs, our engineering team began performing service restarts to process the delayed job queue. This restored functionality for all affected features at 9:25 AM PT on October 18th.
12:27 PM PST
Thanks for your patience. Notifications, unfurling links, loading threads, and our Events API should be back to normal now. If you're still seeing issues, drop us a line at feedback@slack.com.
10:05 AM PST
We're in the final steps to restore full service to the Events API.
9:59 AM PST
Our team is close to a resolution with our Events API. Your continued patience is appreciated.
9:29 AM PST
Events API is delayed and we're working to restore the service as quick as we can. Apologies for the trouble.
8:59 AM PST
We're seeing improvements with most features working correctly again. Delays are still affecting the Events API and we're working to fix it - appreciate the patience.
8:30 AM PST
We're still patching up the delays affecting notifications, unfurling links, and loading threads. Apologies for the problems.
7:56 AM PST
Notifications, threads, and unfurling are not working correctly for some users. We're working to bring back full functionality.
7:27 AM PST
We're still investigating delays and problems with some features, including unfurling, threads, and notifications.
6:56 AM PST
Some users might be running into problems with processing threads and unfurls. Notifications might be delayed as well. We're sorry about the problem and working to fix it.
6:26 AM PST
Services affected
Link Previews
Notifications
Apps/Integrations/APIs
Status
Incident