Tuesday February 19, 2019

Incident

Trouble with sending messages

Issue Summary:

On February 19th at 3:00PM PT, we deployed a change to one of our core systems. At 3:03PM PT we received internal warnings that this change had caused elevated error rates in message sending and began investigating. By 4:39PM PT we had resolved the issue and all messages were sending as expected.

11:38 PM PST

The team have been monitoring things and have given us the all-clear. If you are still experiencing issues with message delivery please let us know at feedback@slack.com. So sorry for all this hassle.

4:59 PM PST

We're investigating why some people are having issues sending messages. Apologies for this disruption.

4:29 PM PST

Services affected

Messaging

Status

Resolved

Outage

Issues with mobile notifications for some Grid customers

Technical Summary:

On February 15th at 12:30 PM PT, a database change was deployed to help us move to a new system. The change in systems resulted in push notifications not being delivered for some Grid customers. Between February 15th and 19th, we received reports of missing notifications from Grid customers which were attributed to a bug and further investigated. On February 19th, our team identified the root issue affecting Grid customers and reverted the deploy. By 6:30 PM PT the revert had been fully deployed and push notifications were working normally again.

We’re very sorry to those affected for the disruption. We’re investigating the root cause and making plans to ensure this doesn’t happen in the future.

11:02 AM PST

We have resolved the notification issue and everyone should be receiving mobile notifications as expected once more. We know this was really disruptive and frustrating, and we're very sorry for the trouble. Thank you for sticking with us.

If you have any further hassles, please reach out to us at feedback@slack.com.

6:44 PM PST

We're on the case and working hard to resolve the notification issue, but we're not out of the woods just yet.

6:36 PM PST

We don't have any news to share just yet, but this is our top priority right now. We appreciate your patience and understanding.

6:06 PM PST

We are still working hard to resolve the issue with mobile notifications.

5:36 PM PST

We have all hands on deck as we continue digging into the mobile notification issue. Thank you for bearing with us.

5:06 PM PST

We are investigating issues with some mobile notifications failing for some Grid customers. We're very sorry for the disruption, and we're working on it.

4:34 PM PST

Services affected

Notifications

Status

Resolved