Thursday March 5, 2020

Issue Summary:

On March 5, 2020 from 5:15 p.m. PST until 7:44 p.m. PST, some customers with Enterprise Key Management (EKM) enabled experienced issues accessing Slack or sending messages. A routine rollback of a previous deployment exposed a flaw in the rollback mechanism, inadvertently deploying a misconfigured artifact to the EKM hosts. This caused connection issues for customers using EKM. We discovered the flaw in the rollback mechanism and redeployed a previous version of the artifact to resolve the issue for all customers.

10:03 PM PST

We have fully resolved the issues with accessing Slack and sending messages. Apologies again for the disruption to your day. Thank you for being patient with us.

7:49 PM PST

We have pushed out a fix and we are monitoring the situation to make sure everything is running smoothly. We'll provide another update once we confirm that the issue is fully resolved.

7:34 PM PST

We've identified the cause of the issue and made some changes to get things back to normal as soon as possible. Most customers should already be seeing improvements.

7:04 PM PST

Our investigation continues, we will be back with more information in 30 minutes. Thank you for bearing with us

6:35 PM PST

Some Grid customers, particularly those with EKM enabled, may be experiencing issues accessing Slack and sending messages. We're very sorry for the disruption and we're on the case.

6:02 PM PST