Friday September 7, 2018

Technical Summary:
On September 7th at 2:35pm PT, we received reports that emails were failing to deliver to Slack forwarding addresses. We identified that this was the result of an expired certificate used to verify requests sent from our email provider. At 4:55pm PT, we deployed an update that corrected this and fixed the problem. Unfortunately any email sent to a forwarding address during this time is not retrievable and will need to be resent.

6:34 PM PST

The issue with Slack's forwarding email feature has been fixed. Thank you for your patience while we resolved this!

4:55 PM PST

A problem with an upstream provider has caused the feature that allows you to send emails to Slack to fail. We’re working with our partners to resolve this problem.

2:35 PM PST

Technical Summary:
On September 7th between 7:43am PT and 8:05am PT, Slack could not connect to our servers in the Sao Paulo region due to network issues. As a result, some people could not access their workspaces.

At 8:05am PT we deployed a fix that would route users out of the troubled region and to a healthier server, which allowed people to successfully connect to Slack again.

We are currently hard at work on scoping preventive measures to ensure this doesn’t happen in the future.

6:23 PM PST

We've patched up the connectivity issues for the South America folks, and things should be working better now with a refresh (CMD/CTRL + R). If this isn't the case, don't hesitate to drop us a line and let us know.

8:23 AM PST

We've confirmed that folks are running into issues when trying to connect from South America specifically. We're looking into this further, and will keep updating with new information.

7:58 AM PST

Technical Summary:
From 11:15am to 11:28am PT, 100% of customers were not able to use Slack.

A deploy to our webapp to enable a feature for how we handle incoming connection requests referenced a function which wasn’t present in a specific file. This missing function caused all incoming requests to fail, returning connection errors to our users.

We rolled back that change for all users at 11:26am. Users were able to receive responses from our app again by 11:28am.

6:20 PM PST

Everyone should be able to connect to their workspace again now. We're very sorry for the disruption to your day, and appreciate your patience while we got things fixed up on our end.

11:28 AM PST

We're investigating problems with connectivity at this time. We're sorry for the interruption and will keep you posted as soon as we have an update.

11:15 AM PST

Technical Summary:
On September 4th at 10:39am PT, we deployed a code change intended to fix an existing bug with guests on Enterprise Grid. On September 7th at 4:49am PT, we first received reports that Enterprise Grid users were unable to send direct messages with users on other workspaces whom they had not messaged before. At 6:38am PT, we reverted that code change and this fixed the problem.

5:59 PM PST

We confirmed the fix. Direct messages are working again for Guests on Enterprise Grid. Apologies for the trouble and thanks for your patience.

7:38 AM PST

We're almost out of the woods. Guests on Enterprise Grid should be able to start new direct messages again. Please reload (Ctrl + R or Cmd + R) to make sure the fix is applied.

7:06 AM PST

We found the cause of the problem with new direct messages for Guests on Enterprise Grid. The team is patching it up and the functionality should be back soon.

6:34 AM PST

We've received reports that some Guests on Enterprise Grid workspaces cannot start new direct messages. Apologies for the trouble - our team is on the case.

4:49 AM PST