Monday October 22, 2018

Technical Summary:

On October 22nd, at approximately 8:37 PM PT, we experienced a small network issue which caused one of our servers to disconnect. This affected the queue that hosts user actions and core functionalities, ranging from push notifications to managing team members. A small subset of our customers experienced degraded functionality during this time.

Our team worked to ensure that we had a plan to resolve the issue, and this was completed by 10:50 PM PT on October 23rd. We continued to monitor the situation, and finally declared the incident fully clear at 11:34 PM PT.

We’re investigating the root cause, as well as making plans to ensure this doesn’t happen in the future.

2:25 PM PST

Thanks for your patience and understanding while we worked to resolve issues with loading new messages. Everyone affected should now be able to access their messages as usual, but please let us know if you run into any further problems at feedback@slack.com.

12:24 AM PST

We're currently experiencing some intermittent degraded functionality, which may cause a very small number of users to have trouble loading new messages in Slack. We're sorry for the disruption, and hope to have things back to normal shortly.

8:37 PM PST

Technical Summary:

At 12:28pm (PT) on October 22, 2018, a change was deployed that caused the hosts responsible for notifications, link unfurls and other tasks to fail. As this change propagated out, customers began experiencing degraded service in the form of late notifications and elevated API response times. After identifying the issue, we updated the affected hosts at 1:13pm (PT), allowing the system to recover and begin processing these tasks again normally. Service was fully restored by 2:40pm (PT).

4:33 PM PST

We’re happy to report that the delays in services have been fixed. We’re sorry for the trouble, and thanks for sticking with us.

3:00 PM PST

We’re happy to report that the delays in services have been fixed. We’re sorry for the trouble, and thanks for sticking with us.

2:59 PM PST

We're seeing a return to normal service but some actions are still being delayed. We'll update you when we're back in business.

2:30 PM PST

Nothing new to share just yet, but we're working to get things back to normal as soon as possible. Again, we truly appreciate your patience.

2:00 PM PST

We're still trying to pin down the root cause of the notifications issue. We've heard reports of other features being affected so we're looking into those reports. We appreciate your patience.

1:30 PM PST

Some users aren't receiving notifications.

1:00 PM PST

Technical Summary:

On October 22nd, 2018 between 8:38 AM (PT) and 9:31 AM (PT), 12% of customers experienced slow searches in Slack, and about 300 customers experienced failed searches.

This outcome was a result of a hardware failure. Once we inspected and removed the problematic hardware, normal search performance resumed for those users.

We are currently hard at work on scoping preventive measures to ensure this doesn’t happen in the future.

4:24 PM PST

Search was degraded for some customers for a short period of time. The issue has been corrected and the team is investigating the cause.

8:38 AM PST