Monday November 19, 2018

Incident

Some customers are unable to use search in their Administration pages

Technical Summary:

On November 19th at approximately 07:46 AM PT our team internally identified a server that manages user-related information which was no longer serving requests in a timely manner. This affected a subset of users, and caused the Admin pages to not load properly. As a result, customers were not able to manage team members. We received the first reports from our customers at 08:54am PT, during this time our team was actively investigating the root cause of the issue.

A few contributing factors led to this incident, including a new feature that relied on the fetching and loading of user-groups and how it the functioned with the auto-complete modal. Our team worked to reduce the strain on the server which hosts user-related information. From there, we were able to deploy a fix, stabilizing the affected server at 04:07 PM PT. An official All-Clear was called at 04:45 PM PT on November 20th.

We’re sorry for the disruption to your work day. If you are still experiencing issues with managing team members, please drop us a line at feedback@slack.com

9:35 AM PST

All features of the Administration pages, including search, should now be working properly. Please get in touch with us directly if any issues persist. We sincerely apologize for any inconvenience this may have caused.

9:48 AM PST

We're seeing more evidence of search returning on the Admin pages for affected customers though we are still working to make sure this is completely resolved before calling the all-clear. Thank you for all your patience through this.

10:31 PM PST

We've been monitoring things and can confirm that everyone should be able to load the Admin page and search User Groups once more. In saying this, search on the Manage members page is slowly recovering and we're making sure search is working generally across the Admin pages. We will share any new information as it becomes available on these outstanding issues.

6:55 PM PST

We're still working on returning Administration page features to full functionality. Please hold tight.

5:27 PM PST

Our team is close to a full resolution, thanks for hanging in there!

4:26 PM PST

No new updates. Affected features are slowly returning but not fully functional just yet.

4:00 PM PST

No new updates. Affected features are slowly returning but not fully functional just yet.

3:56 PM PST

We’re almost out of the woods. The ability to load the Administration page and search for User Groups is slowly returning, although some users may have difficulty searching for members.

3:26 PM PST

We are getting reports of some users being able to reload the Administration page again. We are still experiencing issues with the ability to search for User Groups. Thanks for hanging in there.

2:57 PM PST

No new updates. We’re still digging into the root cause of issues with the Administrator page and the ability to search for User Groups.

2:25 PM PST

We are getting reports that the ability to search for user groups has also been affected. Apologies for the continued interruption to your day.

1:55 PM PST

We have no new information to share just yet, but we're continuing our efforts. We truly appreciate your patience.

1:29 PM PST

We're getting closer to a resolution, but some users may still see problems loading their Manage Members (Administration) page. Thanks for hanging in there.

12:54 PM PST

No news to share at this time, but our efforts to remedy this are ongoing.

12:24 PM PST

We're continuing to investigate a solution for the customers who are unable to load the Administration page.

11:50 AM PST

Our team is still investigating, but we’re on the right track towards a solution. We hope to have more good news to share shortly.

11:21 AM PST

We don't have any further updates right now, but thank you for your patience while we continue to investigate.

10:50 AM PST

We're still trying to pin down the root cause of the Administration page failing to load for some customers. We hope to bring some good news soon. We appreciate your patience.

10:21 AM PST

We are continuing to investigate issues with loading the Administration page for some customers.

9:49 AM PST

Some folks are having troubles loading their "Manage members" administration page. We're looking into the root cause.

9:16 AM PST

Services affected

Workspace/Org Administration

Status

Incident

Incident

Connectivity troubles

Technical Summary:

On November 19, 2018 between 05:12 am PT and 06:00 am PT, a small percentage of our users were unable to connect to Slack.

The connection issue was caused when a single Point of Presence (POP) failed to respond to requests to slack.com. Requests going through that specific POP were either delayed or timed out, which resulted in our users being unable to connect to their workspace, or experiencing slowness in Slack. We worked with our cloud provider to remove this POP from our distribution, and resolved the issue at 06:08 am.

7:15 PM PST

We've got good news: everyone should now be able to connect again! Please reload Slack using Ctrl or Cmd + R to reconnect to your workspace, and let us know at feedback@slack.com if you're still seeing issues after that.

6:08 AM PST

We are getting reports that folks cannot connect to Slack and that some of our services are slow. We are investigating these issues as we speak, and we thank you for your patience.

5:12 AM PST

Services affected

Connectivity

Status

Incident