Wednesday February 11, 2015

At last, we believe we can sound the all-clear. All of the issues people have experienced editing messages (including the misleading error message), receiving messages from integrations, and viewing message history have been resolved.

The underlying issue is a cluster of servers that were exhausting all of the network resources available to them. We're taking several steps now to be better able to operate under these limits and are modifying our capacity planning processes to avoid coming too close to these limits in the future.

5:33 PM PST

We believe almost all of the issues people have experienced editing messages (including the misleading error message) and receiving messages from integrations have been resolved. We're still working to make absolutely sure all our users are able to see all of their messages.

The underlying issue is a cluster of servers that were exhausting all of the network resources available to them. We're taking several steps now to be better able to operate under these limits and are modifying our capacity planning processes to avoid coming too close to these limits in the future.

3:16 PM PST

We're making slow but steady progress on this morning's (and now afternoon's) issues which are causing message edits to fail for some users with a misleading error message (your message was not deleted but your edit was unfortunately lost), messages from integrations to be slow to appear, missing messages when loading history and the very occasional failure to connect to Slack (which can be fixed simply by retrying).

We'll continue to post periodic updates as we recover and take steps to prevent this unacceptably bad experience from re-occuring. Thank you for your patience.

1:11 PM PST

We're making slow but steady progress on this morning's (and now afternoon's) issues which are causing message edits to fail for some users with a misleading error message (your message was not deleted but your edit was unfortunately lost), messages from integrations to be slow to appear, and the very occasional failure to connect to Slack (which can be fixed simply by retrying).

We'll continue to post periodic updates as we recover and take steps to prevent this unacceptably bad experience. Thank you for your patience.

12:53 PM PST

We're still working as quickly as we can to resolve two issues that are the results of overload to our main databases. The first is that if you edit a message it may fail claiming the message has been deleted; your message is safe but the edit will not succeed. The second is that many integrations will be slow to respond until we fix this problem.

11:21 AM PST

Users who edit a message quickly after sending it may get an error that the message was deleted. This error is incorrect — the message is not deleted, but the edit will be lost.

Additionally, some integrations are slow to post to channels.

We're aware of both issues and are working on a fix.

8:51 AM PST