Monday November 19 - Wednesday November 21, 2018

Incident

Some customers are unable to use search in their Administration pages

Technical Summary:

On November 19th at approximately 07:46 AM PT our team internally identified a server that manages user-related information which was no longer serving requests in a timely manner. This affected a subset of users, and caused the Admin pages to not load properly. As a result, customers were not able to manage team members. We received the first reports from our customers at 08:54am PT, during this time our team was actively investigating the root cause of the issue.

A few contributing factors led to this incident, including a new feature that relied on the fetching and loading of user-groups and how it the functioned with the auto-complete modal. Our team worked to reduce the strain on the server which hosts user-related information. From there, we were able to deploy a fix, stabilizing the affected server at 04:07 PM PT. An official All-Clear was called at 04:45 PM PT on November 20th.

We’re sorry for the disruption to your work day. If you are still experiencing issues with managing team members, please drop us a line at feedback@slack.com

9:35 AM PST - Nov 21st・See in your timezone

All features of the Administration pages, including search, should now be working properly. Please get in touch with us directly if any issues persist. We sincerely apologize for any inconvenience this may have caused.

9:48 AM PST - Nov 20th・See in your timezone

We're seeing more evidence of search returning on the Admin pages for affected customers though we are still working to make sure this is completely resolved before calling the all-clear. Thank you for all your patience through this.

10:31 PM PST - Nov 19th・See in your timezone

We've been monitoring things and can confirm that everyone should be able to load the Admin page and search User Groups once more. In saying this, search on the Manage members page is slowly recovering and we're making sure search is working generally across the Admin pages. We will share any new information as it becomes available on these outstanding issues.

6:55 PM PST - Nov 19th・See in your timezone

We're still working on returning Administration page features to full functionality. Please hold tight.

5:27 PM PST - Nov 19th・See in your timezone

Our team is close to a full resolution, thanks for hanging in there!

4:26 PM PST - Nov 19th・See in your timezone

No new updates. Affected features are slowly returning but not fully functional just yet.

4:00 PM PST - Nov 19th・See in your timezone

No new updates. Affected features are slowly returning but not fully functional just yet.

3:56 PM PST - Nov 19th・See in your timezone

We’re almost out of the woods. The ability to load the Administration page and search for User Groups is slowly returning, although some users may have difficulty searching for members.

3:26 PM PST - Nov 19th・See in your timezone

We are getting reports of some users being able to reload the Administration page again. We are still experiencing issues with the ability to search for User Groups. Thanks for hanging in there.

2:57 PM PST - Nov 19th・See in your timezone

No new updates. We’re still digging into the root cause of issues with the Administrator page and the ability to search for User Groups.

2:25 PM PST - Nov 19th・See in your timezone

We are getting reports that the ability to search for user groups has also been affected. Apologies for the continued interruption to your day.

1:55 PM PST - Nov 19th・See in your timezone

We have no new information to share just yet, but we're continuing our efforts. We truly appreciate your patience.

1:29 PM PST - Nov 19th・See in your timezone

We're getting closer to a resolution, but some users may still see problems loading their Manage Members (Administration) page. Thanks for hanging in there.

12:54 PM PST - Nov 19th・See in your timezone

No news to share at this time, but our efforts to remedy this are ongoing.

12:24 PM PST - Nov 19th・See in your timezone

We're continuing to investigate a solution for the customers who are unable to load the Administration page.

11:50 AM PST - Nov 19th・See in your timezone

Our team is still investigating, but we’re on the right track towards a solution. We hope to have more good news to share shortly.

11:21 AM PST - Nov 19th・See in your timezone

We don't have any further updates right now, but thank you for your patience while we continue to investigate.

10:50 AM PST - Nov 19th・See in your timezone

We're still trying to pin down the root cause of the Administration page failing to load for some customers. We hope to bring some good news soon. We appreciate your patience.

10:21 AM PST - Nov 19th・See in your timezone

We are continuing to investigate issues with loading the Administration page for some customers.

9:49 AM PST - Nov 19th・See in your timezone

Some folks are having troubles loading their "Manage members" administration page. We're looking into the root cause.

9:16 AM PST - Nov 19th・See in your timezone

Services affected

Workspace/Org Administration

Status

Resolved