Monday November 19 - Saturday August 24, 2019


Some customers are unable to use search in their Administration pages

Technical Summary:

On November 19th at approximately 07:46 AM PT our team internally identified a server that manages user-related information which was no longer serving requests in a timely manner. This affected a subset of users, and caused the Admin pages to not load properly. As a result, customers were not able to manage team members. We received the first reports from our customers at 08:54am PT, during this time our team was actively investigating the root cause of the issue.

A few contributing factors led to this incident, including a new feature that relied on the fetching and loading of user-groups and how it the functioned with the auto-complete modal. Our team worked to reduce the strain on the server which hosts user-related information. From there, we were able to deploy a fix, stabilizing the affected server at 04:07 PM PT. An official All-Clear was called at 04:45 PM PT on November 20th.

We’re sorry for the disruption to your work day. If you are still experiencing issues with managing team members, please drop us a line at

9:35 AM PST

All features of the Administration pages, including search, should now be working properly. Please get in touch with us directly if any issues persist. We sincerely apologize for any inconvenience this may have caused.

9:48 AM PST

We're seeing more evidence of search returning on the Admin pages for affected customers though we are still working to make sure this is completely resolved before calling the all-clear. Thank you for all your patience through this.

10:31 PM PST

We've been monitoring things and can confirm that everyone should be able to load the Admin page and search User Groups once more. In saying this, search on the Manage members page is slowly recovering and we're making sure search is working generally across the Admin pages. We will share any new information as it becomes available on these outstanding issues.

6:55 PM PST

We're still working on returning Administration page features to full functionality. Please hold tight.

5:27 PM PST

Our team is close to a full resolution, thanks for hanging in there!

4:26 PM PST

No new updates. Affected features are slowly returning but not fully functional just yet.

4:00 PM PST

No new updates. Affected features are slowly returning but not fully functional just yet.

3:56 PM PST

We’re almost out of the woods. The ability to load the Administration page and search for User Groups is slowly returning, although some users may have difficulty searching for members.

3:26 PM PST

We are getting reports of some users being able to reload the Administration page again. We are still experiencing issues with the ability to search for User Groups. Thanks for hanging in there.

2:57 PM PST

No new updates. We’re still digging into the root cause of issues with the Administrator page and the ability to search for User Groups.

2:25 PM PST

We are getting reports that the ability to search for user groups has also been affected. Apologies for the continued interruption to your day.

1:55 PM PST

We have no new information to share just yet, but we're continuing our efforts. We truly appreciate your patience.

1:29 PM PST

We're getting closer to a resolution, but some users may still see problems loading their Manage Members (Administration) page. Thanks for hanging in there.

12:54 PM PST

No news to share at this time, but our efforts to remedy this are ongoing.

12:24 PM PST

We're continuing to investigate a solution for the customers who are unable to load the Administration page.

11:50 AM PST

Our team is still investigating, but we’re on the right track towards a solution. We hope to have more good news to share shortly.

11:21 AM PST

We don't have any further updates right now, but thank you for your patience while we continue to investigate.

10:50 AM PST

We're still trying to pin down the root cause of the Administration page failing to load for some customers. We hope to bring some good news soon. We appreciate your patience.

10:21 AM PST

We are continuing to investigate issues with loading the Administration page for some customers.

9:49 AM PST

Some folks are having troubles loading their "Manage members" administration page. We're looking into the root cause.

9:16 AM PST

Services affected

Workspace/Org Administration