Monday November 2, 2020

Issue summary:
Between November 2, 2020 at 10:58 a.m. PST and November 4, 2020 at 4:55 p.m. PST, four separate spikes in errors caused some users to experience failures in search functionality. These failures included seeing a "Something went wrong" error message when searching for files or seeing sub-optimal search results.

We determined that a lack of utilization and retry limits that were set on searches caused the search service to initially overload for 1 hour at 10:58 a.m. PST on November 2, 2020.

We restored search functionality, but the underlying service failed to fully recover. This left the search service vulnerable to disruption from future complex queries and caused search failures again on November 3 and November 4.

We updated our configurations to reduce the allocated memory for searches which allowed full recovery of search services.

2:21 PM PST

Recent updates (such as new messages) should now appear in search results as expected. Thank you for your patience while we resolved this issue and we'll be back with an issue summary.

6:00 PM PST

While affected users should still be able to search in Slack, recent updates, such as new messages, may not immediately appear in the results. We'll let you know once the issue is resolved, and we'll follow-up if the situation changes.

1:50 PM PST

Affected users should be able to search and see the most optimal results. However, we're still working on this issue. We'll let you know if the situation changes, and we'll follow-up once this issue is resolved.

1:02 PM PST

The majority of affected users should be able to search in Slack. Some users may not see the most optimal search results, and some users may see a "Something went wrong" error when searching for files. We'll continue to work to resolve this trouble.

Note: We updated this note to mention some users could've encountered a "Something went wrong" error when searching for files, so all affected users weren't able to search in Slack at this time.

12:16 PM PST

At this time, some users may be unable to search for messages and files specifically. We're working to resolve this issue, and we appreciate your patience.

11:43 AM PST

Some users are unable to search in Slack. We're investigating, and we'll follow-up here once we know more. We're sorry for the trouble.

11:13 AM PST