Monday October 22, 2018

Incident

Degraded functionality

Technical Summary:

On October 22nd, at approximately 8:37 PM PT, we experienced a small network issue which caused one of our servers to disconnect. This affected the queue that hosts user actions and core functionalities, ranging from push notifications to managing team members. A small subset of our customers experienced degraded functionality during this time.

Our team worked to ensure that we had a plan to resolve the issue, and this was completed by 10:50 PM PT on October 23rd. We continued to monitor the situation, and finally declared the incident fully clear at 11:34 PM PT.

We’re investigating the root cause, as well as making plans to ensure this doesn’t happen in the future.

2:25 PM PDT - Oct 25th・See in your timezone

Thanks for your patience and understanding while we worked to resolve issues with loading new messages. Everyone affected should now be able to access their messages as usual, but please let us know if you run into any further problems at feedback@slack.com.

12:24 AM PDT - Oct 24th・See in your timezone

We're currently experiencing some intermittent degraded functionality, which may cause a very small number of users to have trouble loading new messages in Slack. We're sorry for the disruption, and hope to have things back to normal shortly.

8:37 PM PDT - Oct 22nd・See in your timezone

Services affected

Apps/Integrations/APIs

Connections

Status

Resolved

Incident

Slack is experiencing degraded service

Technical Summary:

At 12:28pm (PT) on October 22, 2018, a change was deployed that caused the hosts responsible for notifications, link unfurls and other tasks to fail. As this change propagated out, customers began experiencing degraded service in the form of late notifications and elevated API response times. After identifying the issue, we updated the affected hosts at 1:13pm (PT), allowing the system to recover and begin processing these tasks again normally. Service was fully restored by 2:40pm (PT).

4:33 PM PDT - Oct 23rd・See in your timezone

We’re happy to report that the delays in services have been fixed. We’re sorry for the trouble, and thanks for sticking with us.

3:00 PM PDT - Oct 22nd・See in your timezone

We’re happy to report that the delays in services have been fixed. We’re sorry for the trouble, and thanks for sticking with us.

2:59 PM PDT - Oct 22nd・See in your timezone

We're seeing a return to normal service but some actions are still being delayed. We'll update you when we're back in business.

2:30 PM PDT - Oct 22nd・See in your timezone

Nothing new to share just yet, but we're working to get things back to normal as soon as possible. Again, we truly appreciate your patience.

2:00 PM PDT - Oct 22nd・See in your timezone

We're still trying to pin down the root cause of the notifications issue. We've heard reports of other features being affected so we're looking into those reports. We appreciate your patience.

1:30 PM PDT - Oct 22nd・See in your timezone

Some users aren't receiving notifications.

1:00 PM PDT - Oct 22nd・See in your timezone

Services affected

Messaging

Link Previews

Posts/Files

Notifications

Apps/Integrations/APIs

Connections

Status

Resolved

Incident

Search returning partial results for some customers

Technical Summary:

On October 22nd, 2018 between 8:38 AM (PT) and 9:31 AM (PT), 12% of customers experienced slow searches in Slack, and about 300 customers experienced failed searches.

This outcome was a result of a hardware failure. Once we inspected and removed the problematic hardware, normal search performance resumed for those users.

We are currently hard at work on scoping preventive measures to ensure this doesn’t happen in the future.

4:24 PM PDT - Oct 23rd・See in your timezone

Search was degraded for some customers for a short period of time. The issue has been corrected and the team is investigating the cause.

8:38 AM PDT - Oct 22nd・See in your timezone

Services affected

Search

Status

Resolved