Tuesday February 19, 2019
Incident
Trouble with sending messages
Issue Summary:
On February 19th at 3:00PM PT, we deployed a change to one of our core systems. At 3:03PM PT we received internal warnings that this change had caused elevated error rates in message sending and began investigating. By 4:39PM PT we had resolved the issue and all messages were sending as expected.
11:38 PM PST
The team have been monitoring things and have given us the all-clear. If you are still experiencing issues with message delivery please let us know at feedback@slack.com. So sorry for all this hassle.
4:59 PM PST
We're investigating why some people are having issues sending messages. Apologies for this disruption.
4:29 PM PST
Services affected
Messaging
Status
Incident
Outage
Issues with mobile notifications for some Grid customers
Technical Summary:
On February 15th at 12:30 PM PT, a database change was deployed to help us move to a new system. The change in systems resulted in push notifications not being delivered for some Grid customers. Between February 15th and 19th, we received reports of missing notifications from Grid customers which were attributed to a bug and further investigated. On February 19th, our team identified the root issue affecting Grid customers and reverted the deploy. By 6:30 PM PT the revert had been fully deployed and push notifications were working normally again.
We’re very sorry to those affected for the disruption. We’re investigating the root cause and making plans to ensure this doesn’t happen in the future.
11:02 AM PST
We have resolved the notification issue and everyone should be receiving mobile notifications as expected once more. We know this was really disruptive and frustrating, and we're very sorry for the trouble. Thank you for sticking with us.
If you have any further hassles, please reach out to us at feedback@slack.com.
6:44 PM PST
We're on the case and working hard to resolve the notification issue, but we're not out of the woods just yet.
6:36 PM PST
We don't have any news to share just yet, but this is our top priority right now. We appreciate your patience and understanding.
6:06 PM PST
We are still working hard to resolve the issue with mobile notifications.
5:36 PM PST
We have all hands on deck as we continue digging into the mobile notification issue. Thank you for bearing with us.
5:06 PM PST
We are investigating issues with some mobile notifications failing for some Grid customers. We're very sorry for the disruption, and we're working on it.
4:34 PM PST
Services affected
Notifications
Status
Outage