https://status.slack.comSlack System Status2024-03-19T00:06:34-07:00Slack
https://slack-status.com/2024-03/f050004edd32fdb2
2024-03-19T00:06:34-07:002024-03-19T00:06:34-07:00Incident: Issues with file previews
Customers on workspaces enrolled in an International Data Residency region may be experiencing blurred file previews on desktop. We have resolved the underlying issue, and we're now working to restore the affected file previews. We'll provide an update once we have more details to share.
<p>
<small>12:06am PDT</small>
Customers on workspaces enrolled in an International Data Residency region may be experiencing blurred file previews on desktop. We have resolved the underlying issue, and we're now working to restore the affected file previews. We'll provide an update once we have more details to share.
</p>
https://slack-status.com/2024-03/d38973d5b190a450
2024-03-18T12:34:39-07:002024-03-18T16:29:50-07:00Incident: Issues with accessing External Connections
Issue summary:
From 1:00 PM PDT on March 16, 2024 to 2:57 PM PDT on March 18, 2024, some users encountered a REB434A8313F error message when attempting to access the External Connections tab in Slack. On March 16th, a small number of users reported impact, however, it was later determined that more users were affected.
We determined that a recent backend change to the navigation bar badging rules was preventing this view from loading properly. We reverted this change, which resolved the issue for all affected users.
<p>
<small>12:34pm PDT</small>
Some users may be seeing a REB434A8313F error message when attempting to access the External Connections tab in their sidebar via More > External. We are currently investigating and believe we have found the root cause.
We will update as we have more information. Thank you for you patience.
</p>
<p>
<small>2:51pm PDT</small>
We have made a change that fixes this issue. Please refresh the app, then navigate to More > External Connections in your tab bar to make any connections. Thank you for your patience.
</p>
<p>
<small>4:29pm PDT</small>
Issue summary:
From 1:00 PM PDT on March 16, 2024 to 2:57 PM PDT on March 18, 2024, some users encountered a REB434A8313F error message when attempting to access the External Connections tab in Slack. On March 16th, a small number of users reported impact, however, it was later determined that more users were affected.
We determined that a recent backend change to the navigation bar badging rules was preventing this view from loading properly. We reverted this change, which resolved the issue for all affected users.
</p>
https://slack-status.com/2024-03/070ad43095b540ab
2024-03-13T14:37:05-07:002024-03-14T18:35:34-07:00Incident: Reports of slowness and connectivity issues in Slack
Issue summary:
From 12:40 PM to 4:55 PM PDT on March 13, 2024, users from one Enterprise Grid organization may have experienced slowness or connectivity issues when using Slack.
A routine infrastructure process inadvertently caused an increase in requests to the database hosting the affected Enterprise Grid organization. This resulted in slower than usual response times for user actions in Slack, and some connectivity failures.
We identified the cause of the problem and manually increased the database capacity, restoring normal functionality for impacted users.
Thank you for bearing with us while we resolved this issue.
<p>
<small>2:37pm PDT</small>
Some customers are encountering slow load times and connectivity errors in Slack. We're actively looking into this, and we'll report back as more information becomes available.
</p>
<p>
<small>4:03pm PDT</small>
Upon investigation we've determined these connectivity problems are isolated to one Enterprise Grid Org.
We'll closely monitor reports and provide further updates through customer support tickets with the impacted users. We will not be sharing further details on this through our Status Site, so please reach out to our team at feedback@slack.com if you need further assistance.
</p>
<p>
<small>6:35pm PDT</small>
Issue summary:
From 12:40 PM to 4:55 PM PDT on March 13, 2024, users from one Enterprise Grid organization may have experienced slowness or connectivity issues when using Slack.
A routine infrastructure process inadvertently caused an increase in requests to the database hosting the affected Enterprise Grid organization. This resulted in slower than usual response times for user actions in Slack, and some connectivity failures.
We identified the cause of the problem and manually increased the database capacity, restoring normal functionality for impacted users.
Thank you for bearing with us while we resolved this issue.
</p>
https://slack-status.com/2024-03/e954ecf8fbab9711
2024-03-13T20:42:22-07:002024-03-14T04:01:37-07:00Incident: User are receiving errors when attempting to access certain app configuration pages.
Issue summary:
On March 13, 2024 from 5:02 PM PDT until 9:48 PM PDT some users may have encountered "Something's gone awry" or "Slack isn't loading" errors when attempting to access their certain app configuration pages.
We determined that a recent code change caused this issue. We reverted the code change and the issue was resolved.
<p>
<small>8:42pm PDT</small>
Some customers are reporting they cannot access certain app configuration pages. They receive "Something’s gone awry" or "Slack isn’t loading" errors. Reports have shown these errors are appearing on the 'Collaborators' or 'Manage Distribution' pages.
We're actively looking into this, and we'll report back here when more information becomes available to us.
</p>
<p>
<small>11:07pm PDT</small>
We’ve restored full functionality to all app configuration pages for all impacted users. You may need to reload these pages (Command/Ctrl + Shift + R) in order for this fix to be reflected on your end. We’re sorry for the disruption we brought to your day.
</p>
<p>
<small>4:01am PDT</small>
Issue summary:
On March 13, 2024 from 5:02 PM PDT until 9:48 PM PDT some users may have encountered "Something's gone awry" or "Slack isn't loading" errors when attempting to access their certain app configuration pages.
We determined that a recent code change caused this issue. We reverted the code change and the issue was resolved.
</p>
https://slack-status.com/2024-03/811378903bd22810
2024-03-11T15:04:41-07:002024-03-12T13:18:00-07:00Outage: Trouble loading multiple features in Slack
Issue summary:
On March 11, 2024 from 2:43 PM PDT until 3:43 PM PDT some users in workspaces using International Data Residency (IDR) experienced issues with several Slack features, including messages loading or sending, channels appearing blank, canvases loading, huddles connecting, and trouble with some workflows were also noted.
We identified a spike in error rates that was tied to a recent change to a backend service configuration logic. This change resulted in a cache discrepancy between services causing the errors. We rolled back the recent changes which immediately reduced error rates, resolving the issue for affected users. We are working on modifying the service handler in an effort to reduce the likelihood of this occurring in the future.
<p>
<small>3:04pm PDT</small>
We're investigating issues users are having with sending and loading messages, including threads. We'll be back with an update as soon as we know more.
</p>
<p>
<small>3:30pm PDT</small>
We're still working to identify the root cause of the issue, however, this is isolated to workspaces with International Data Residency (IDR) enabled. Affected users on desktop and mobile may be experiencing additional issues with loading channels, huddles, canvases, and some workflows.
</p>
<p>
<small>3:51pm PDT</small>
The cause of the issue has been identified. We rolled back a recent change that resolved the issue for affected users. We appreciate your patience while we sorted this out and apologize for any interruption to your day.
</p>
<p>
<small>1:18pm PDT</small>
Issue summary:
On March 11, 2024 from 2:43 PM PDT until 3:43 PM PDT some users in workspaces using International Data Residency (IDR) experienced issues with several Slack features, including messages loading or sending, channels appearing blank, canvases loading, huddles connecting, and trouble with some workflows were also noted.
We identified a spike in error rates that was tied to a recent change to a backend service configuration logic. This change resulted in a cache discrepancy between services causing the errors. We rolled back the recent changes which immediately reduced error rates, resolving the issue for affected users. We are working on modifying the service handler in an effort to reduce the likelihood of this occurring in the future.
</p>
https://slack-status.com/2024-03/c60bb5e0edd2cd89
2024-03-11T17:03:38-07:002024-03-12T12:21:05-07:00Incident: Issue with focus in workflows
Issue Summary:
On March 11, 2024 at 3:00 PDT to March 12, 2024 at 12:00 AM PDT, some users had trouble completing workflows that contained either text or drop-down fields due to a bug that caused the cursor to revert to the previous field in the workflow.
Upon investigation, we discovered a recent code change that introduced this bug. We rolled back the change and pushed a fix to all users resolving the issue.
Thank you again for bearing with us in the meantime.
<p>
<small>5:03pm PDT</small>
We are currently investigating an issue that is causing focus within workflow forms to erroneously jump back to the first field when entering text in any other field. We'll let you know as soon as we have another update, and we apologize for the hassle in the meantime.
</p>
<p>
<small>5:56pm PDT</small>
We believe we've identified the root cause of the issue, and are rolling back a recent change that will hopefully resolve the issue for those affected.
Some customers may have success in the meantime with filling out workflow forms in a web browser. Our team are still working to fully resolve the issue though, and we'll be back in touch when we have more news to share.
</p>
<p>
<small>8:08pm PDT</small>
We have deployed a fix to resolve the workflow issue, and this update is currently rolling out to everyone.
To fetch the fix immediately, please refresh Slack with Cmd/Ctrl + Shift + R. Slack will reload, but won't sign you out.
Thank you for bearing with us, we appreciate it.
</p>
<p>
<small>12:21pm PDT</small>
Issue Summary:
On March 11, 2024 at 3:00 PDT to March 12, 2024 at 12:00 AM PDT, some users had trouble completing workflows that contained either text or drop-down fields due to a bug that caused the cursor to revert to the previous field in the workflow.
Upon investigation, we discovered a recent code change that introduced this bug. We rolled back the change and pushed a fix to all users resolving the issue.
Thank you again for bearing with us in the meantime.
</p>
https://slack-status.com/2024-03/1f506e0a906e75bb
2024-03-06T19:59:54-08:002024-03-08T06:35:09-08:00Incident: Issues with legacy custom integration configuration pages
Issue summary:
On March 6, 2024 from 7:45 PM PST to around 9:33 PM PST customers reported that they were receiving a 500 error when viewing the legacy Incoming Webhook custom integration page.
We identified a backend code change that inadvertently changed data that the Incoming Webhook page needed to load which consequently caused the page to respond with an HTTP 500 error.
We rolled back the change, correcting the issue and all customers should now be able to access the Incoming Webhook page as expected.
<p>
<small>7:59pm PST</small>
Some customers may be experiencing issues accessing the Incoming WebHooks legacy custom integration page. We're aware and investigating the problem.
We apologize for the disruption. We'll provide an update as soon as we have news to share.
</p>
<p>
<small>9:40pm PST</small>
We’ve released a fix and all impacted pages are up and running once again. You may need to reload (Command/Ctrl + Shift + R) any affected pages in your web browser in order for this fix to be reflected on your end. We appreciate your patience while we sorted this out.
</p>
<p>
<small>6:35am PST</small>
Issue summary:
On March 6, 2024 from 7:45 PM PST to around 9:33 PM PST customers reported that they were receiving a 500 error when viewing the legacy Incoming Webhook custom integration page.
We identified a backend code change that inadvertently changed data that the Incoming Webhook page needed to load which consequently caused the page to respond with an HTTP 500 error.
We rolled back the change, correcting the issue and all customers should now be able to access the Incoming Webhook page as expected.
</p>
https://slack-status.com/2024-03/b9c8573e14eec256
2024-03-06T18:41:36-08:002024-03-08T02:54:49-08:00Incident: Huddles not working for some Windows users
Issue summary:
On March 5, 2024 from 6:05 PM PST to March 6, 2024 8:17 AM PST customers on the Windows operating systems reported that Huddles were freezing when screen sharing.
We determined that a recent change that was made to the video codec of Huddles, which may have been moving more of the video encoding to the computer's Graphic Processing Unit. This would cause performance problems & freezing during a screen share.
After narrowing down the exact cause we rolled back the change which restored full functionality to Huddles screen sharing.
<p>
<small>6:41pm PST</small>
We’re investigating a problem with huddles. We’ll be back with another update as soon as we have more information to share.
</p>
<p>
<small>9:33pm PST</small>
We’ve rolled out a fix and huddles are now working as expected. If you’re still encountering any trouble, please reload Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux). Thank you for your patience while we sorted out this issue.
</p>
<p>
<small>2:54am PST</small>
Issue summary:
On March 5, 2024 from 6:05 PM PST to March 6, 2024 8:17 AM PST customers on the Windows operating systems reported that Huddles were freezing when screen sharing.
We determined that a recent change that was made to the video codec of Huddles, which may have been moving more of the video encoding to the computer's Graphic Processing Unit. This would cause performance problems & freezing during a screen share.
After narrowing down the exact cause we rolled back the change which restored full functionality to Huddles screen sharing.
</p>
https://slack-status.com/2024-03/a20334cecc6fff7c
2024-03-05T20:32:05-08:002024-03-05T23:23:50-08:00Incident: Some customers may be unable to access the App Directory
Issue summary:
On March 5, 2024 from 3:36 PM PST until 9:36 PM PST some customers were unable to visit the app detail page in App Directory, instead seeing a page displaying a "Server Error" message.
We traced the issue to a recent backend code change and reverted the change, which fixed this issue.
<p>
<small>8:32pm PST</small>
Some customers are encountering an error when trying to access the App Directory. We're actively looking into this, and we'll report back as more information becomes available.
</p>
<p>
<small>9:03pm PST</small>
We’re still looking into the cause of these errors, and we’ll provide new information as soon as it’s available.
</p>
<p>
<small>9:37pm PST</small>
We’ve released a fix and the App Directory is up and running once again. We appreciate your patience while we sorted this out.
</p>
<p>
<small>11:23pm PST</small>
Issue summary:
On March 5, 2024 from 3:36 PM PST until 9:36 PM PST some customers were unable to visit the app detail page in App Directory, instead seeing a page displaying a "Server Error" message.
We traced the issue to a recent backend code change and reverted the change, which fixed this issue.
</p>
https://slack-status.com/2024-03/9c34d28b9b3f4e7b
2024-03-04T17:14:32-08:002024-03-05T21:00:19-08:00Incident: 'Invite people' popup menu in Manage Members page populates name of wrong workspace.
Issue summary:
From March 1, 2024 at 12:13 PM PST to March 4, 2024 at 7:06 PM PST, Enterprise Grid admins and owners may have experienced issues inviting members to workspaces.
We made a code change that inadvertently introduced a logic error, resulting in the incorrect workspaces being listed in the invite modal. We reverted the code change, restoring normal invite functionality for all affected users.
<p>
<small>5:14pm PST</small>
Some customers with an admin role in Slack are reporting that when they go to invite new users through the Manage Members page (workspacename.slack.com/admin), they will select ‘Invite people’, but the popup menu displays an irrelevant workspace.
A workaround to this whilst we work on a fix, is to simply use the 'Invite people to' method within the app instead. Customers can click on their workspace name in the top-left of their app and select 'Invite people to'.
We'll be back with another update when we know more, and we apologize for any confusion this issue may cause.
</p>
<p>
<small>12:03pm PST</small>
Workspace Owners and Admins should no longer be experiencing any trouble inviting new users. Apologies for the trouble and thank you for your patience.
</p>
<p>
<small>9:00pm PST</small>
Issue summary:
From March 1, 2024 at 12:13 PM PST to March 4, 2024 at 7:06 PM PST, Enterprise Grid admins and owners may have experienced issues inviting members to workspaces.
We made a code change that inadvertently introduced a logic error, resulting in the incorrect workspaces being listed in the invite modal. We reverted the code change, restoring normal invite functionality for all affected users.
</p>
https://slack-status.com/2024-03/629a8aa472f5f9c2
2024-03-04T20:32:04-08:002024-03-04T20:32:04-08:00Incident: Some customers are unable to share their screen during a huddle
Issue summary:
On March 4, 2024, from 4:18 PM PST until 10:17 PM PST some users experienced an issue when sharing their screen during a huddle. Attempting to share their screen would fail without any error or warning.
An incorrect piece of code was identified as the cause of the screen share issue. We safely reverted the problematic code, resolving the issue that users experienced. To capture the fix, a hard refresh (by pressing Cmd/Ctrl + Shift + R) may be required.
<p>
<small>8:32pm PST</small>
We’re investigating reports that some customers are unable to share their screen during a huddle. We’re on the case and will provide another update once we have more details to share. Thank you for bearing with us in the meantime.
</p>
<p>
<small>9:04pm PST</small>
We’re still investigating issues with screensharing in huddles. No additional information to share just yet, but we’ll keep you posted as soon we have an update.
</p>
<p>
<small>9:36pm PST</small>
We’ve identified the cause of this issue and we’re working on getting a fix out as soon as possible. We’ll be back once we have another update to share.
</p>
<p>
<small>10:21pm PST</small>
We are continuing to work on getting a fix out as quickly as possible. Thank you for your continued patience.
</p>
<p>
<small>12:42am PST</small>
We’ve rolled out the fix and huddle screen sharing is now working as expected. If you’re still encountering any trouble, please reload Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux). Thank you for your patience while we sorted out this issue.
</p>
<p>
<small>9:41am PST</small>
Issue summary:
On March 4, 2024, from 4:18 PM PST until 10:17 PM PST some users experienced an issue when sharing their screen during a huddle. Attempting to share their screen would fail without any error or warning.
An incorrect piece of code was identified as the cause of the screen share issue. We safely reverted the problematic code, resolving the issue that users experienced. To capture the fix, a hard refresh (by pressing Cmd/Ctrl + Shift + R) may be required.
</p>
https://slack-status.com/2024-02/35134eda119581f5
2024-02-29T14:26:48-08:002024-02-29T21:48:33-08:00Incident: Users unable to create or edit workflows
Issue summary:
On February 29, 2024 from 1:28 PM PST to around 4:36 PM PST, users were unable to create or update some workflows.
We determined that a recent code change had inadvertently introduced a logic problem, resulting in errors for workflows that used event triggers. We reverted this code change. However, rolling back reintroduced a separate issue that we'd resolved earlier in the day.
We prepared a tailored revert to roll back the code causing the workflow problem while also including the fix for the previous issue. This second revert resolved both problems, restoring full functionality to Slack.
Thank you for being patient with us.
<p>
<small>2:26pm PST</small>
We're currently investigating an issue that is preventing users from creating and updating workflows. We'll be back with another update when we know more, and we apologize for the inconvenience in the meantime.
</p>
<p>
<small>2:47pm PST</small>
We have determined that this issue is specific to workflows that contain event triggers. We've discovered the root cause and are working to resolve this as quickly as possible. Thank you for bearing with us in the meantime.
</p>
<p>
<small>3:16pm PST</small>
We're still actively working on getting this fixed, but we don’t have any new information to share at this time. Thank you for your continued patience.
</p>
<p>
<small>4:33pm PST</small>
We're continuing to work on restoring workflow functionality for all users.
We initially rolled back what we thought was the problematic code, though this didn't create positive change to the issue at hand.
We're looking into another possible solution and will report back as soon as we have more news to share.
</p>
<p>
<small>4:33pm PST</small>
</p>
<p>
<small>5:08pm PST</small>
We've restored full functionality to the workflow tool for all users.
No steps are required to see this fix in your workspaces.
We're sorry for the disruption this caused.
</p>
<p>
<small>9:48pm PST</small>
Issue summary:
On February 29, 2024 from 1:28 PM PST to around 4:36 PM PST, users were unable to create or update some workflows.
We determined that a recent code change had inadvertently introduced a logic problem, resulting in errors for workflows that used event triggers. We reverted this code change. However, rolling back reintroduced a separate issue that we'd resolved earlier in the day.
We prepared a tailored revert to roll back the code causing the workflow problem while also including the fix for the previous issue. This second revert resolved both problems, restoring full functionality to Slack.
Thank you for being patient with us.
</p>
https://slack-status.com/2024-02/939919184ea7655b
2024-02-29T09:06:07-08:002024-02-29T12:47:52-08:00Incident: Workflows are unable to be viewed or triggered
Issue summary:
On February 29, 2024 at 8:00 AM PST to 9:55 AM PST, users saw errors when attempting to trigger and manage legacy workflows.
Upon investigation, we discovered a code change prevented part of the infrastructure that supports workflows from functioning correctly.
To resolve the issue, we reverted the code change and all workflows were restored.
<p>
<small>9:06am PST</small>
We’re investigating a problem with legacy workflows. We’ll be back with another update as soon as we have more information to share.
</p>
<p>
<small>9:30am PST</small>
We’re still investigating issues with workflows. No additional information to share just yet, but we’ll keep you posted as soon we have an update.
</p>
<p>
<small>10:09am PST</small>
We’ve resolved the issue with legacy workflows and they've been restored to full functionality. Thank you very much for your patience while we worked on getting this fix out the door.
</p>
<p>
<small>12:47pm PST</small>
Issue summary:
On February 29, 2024 at 8:00 AM PST to 9:55 AM PST, users saw errors when attempting to trigger and manage legacy workflows.
Upon investigation, we discovered a code change prevented part of the infrastructure that supports workflows from functioning correctly.
To resolve the issue, we reverted the code change and all workflows were restored.
</p>
https://slack-status.com/2024-02/2ea2fda2874858b7
2024-02-27T21:55:13-08:002024-02-27T21:55:13-08:00Incident: Trouble signing in on mobile for some Owners and Admins
Issue Summary:
On February 27, 2024 from around 8:30 PM PST to 10:06 PM PST Workspace Owners and Admins were experiencing issues while setting up two-factor authentication during sign in on iOS devices.
The issue was caused by a two-factor authentication feature deploy to users on the Pro subscription. After identifying the root cause the feature was was rolled back and this allowed all users to log back into Slack.
<p>
<small>9:55pm PST</small>
Some owners and admins of workspaces on the Pro plan may be unable to sign in to the mobile app at the moment.
We're on the case, and we'll be back with more information in 30 minutes.
If you need urgent assistance signing in to the mobile app, please contact us at feedback@slack.com.
</p>
<p>
<small>10:36pm PST</small>
Customers should no longer be experiencing any trouble signing in on mobile app. Apologies for the trouble today and thank you for your patience.
</p>
<p>
<small>4:26am PST</small>
Issue Summary:
On February 27, 2024 from around 8:30 PM PST to 10:06 PM PST Workspace Owners and Admins were experiencing issues while setting up two-factor authentication during sign in on iOS devices.
The issue was caused by a two-factor authentication feature deploy to users on the Pro subscription. After identifying the root cause the feature was was rolled back and this allowed all users to log back into Slack.
</p>
https://slack-status.com/2024-02/cedce965afd47ebb
2024-02-27T07:41:56-08:002024-02-27T07:41:56-08:00Incident: Close button for right pane not appearing for some users
Issue summary:
On February 26, 2024, from 9:21 AM PST to 9:16 AM PST on February 27, some users encountered the close button disappearing in the right pane where threads and profiles are viewed.
We discovered this was caused by a missing check in the change workspace process that prevented the right pane elements from rendering properly. Once the cause was identified, a change was made to the logic for drawing Slack's interface when switching workspaces. Once the change was rolled out, the issue was resolved.
<p>
<small>7:41am PST</small>
Some customers are reporting they are missing the close button in the right thread pane. Some users have reported that refreshing Slack via Ctrl + R (Windows/Linux) or Cmd + R (macOS) fixes the issue.
We're actively looking into this, and we'll report back as more information becomes available.
</p>
<p>
<small>8:02am PST</small>
We've identified the cause of this issue and we're working on getting a fix out as soon as possible. We'll be back once we have another update to share.
</p>
<p>
<small>9:24am PST</small>
We've rolled out a fix and the right pane is now working as expected. If you're still encountering any trouble, please reload Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux). Thank you for your patience while we sorted out this issue.
</p>
<p>
<small>11:12am PST</small>
Issue summary:
On February 26, 2024, from 9:21 AM PST to 9:16 AM PST on February 27, some users encountered the close button disappearing in the right pane where threads and profiles are viewed.
We discovered this was caused by a missing check in the change workspace process that prevented the right pane elements from rendering properly. Once the cause was identified, a change was made to the logic for drawing Slack's interface when switching workspaces. Once the change was rolled out, the issue was resolved.
</p>
https://slack-status.com/2024-02/b1389c8cd318a7d1
2024-02-26T13:02:33-08:002024-02-26T18:50:48-08:00Incident: Users are not able to create workflows
Issue summary:
From 12:30 PM PST to 1:05 PM PST on February 26, 2024, users were unable to create workflows.
We determined that the related API method needed updating in order to eliminate inconsistencies that were causing elevated replication lag.
We deployed an update to the API method correcting this issue, and all workflow creation will now work as expected.
<p>
<small>1:02pm PST</small>
We are currently investigating an issue where users are not able to create workflows. Running existing workflows is not impacted. We will update as we have more information to share.
</p>
<p>
<small>3:20pm PST</small>
We have isolated the cause of this issue and are working on getting a fix out as soon as we can. We’ll be back as soon as we have another update, and we appreciate your patience in the meantime.
</p>
<p>
<small>3:56pm PST</small>
We've released a fix and users should no longer encounter any issues creating new workflows. Thank you for bearing with us as we worked to get this resolved.
</p>
<p>
<small>6:50pm PST</small>
Issue summary:
From 12:30 PM PST to 1:05 PM PST on February 26, 2024, users were unable to create workflows.
We determined that the related API method needed updating in order to eliminate inconsistencies that were causing elevated replication lag.
We deployed an update to the API method correcting this issue, and all workflow creation will now work as expected.
</p>
https://slack-status.com/2024-02/4c0af99013428a7a
2024-02-26T09:49:25-08:002024-02-26T17:47:26-08:00Incident: Issues sending and loading messages
Issue summary:
On February 26, 2024 from 9:00 AM PST to 9:36 AM PST, some users experienced problems sending and loading messages in Slack. Others may have completed these actions successfully but more slowly than expected.
Earlier today, we deployed a routine update to part of our network infrastructure. During this work, we deprovisioned some of our proxy servers and provisioned a new set. Some of our webapp servers took longer than expected to sync with the new list of proxy servers, resulting in a spike in connection errors. This coincided with the regular 9:00 AM PST increase in traffic to our servers, which may have exacerbated impact.
By around 9:36 AM PST, error rates had decreased and user impact had completely subsided, most likely due to our automatic scaling functionality. However, we're continuing to analyze data from this event to fully understand the root cause and prevent similar issues in the future.
<p>
<small>9:49am PST</small>
Starting at 9:00 AM PST on Monday, February 26 to 9:30 AM PST, a small percentage of users encountered trouble sending messages, loading messages, and accessing Slack.
These errors were caused by an increased load on our servers, and reduced as our servers balanced the load across our system. User should no longer be encountering issues with Slack.
Please send a note to feedback@slack.com if you continue to have any trouble.
</p>
<p>
<small>5:47pm PST</small>
Issue summary:
On February 26, 2024 from 9:00 AM PST to 9:36 AM PST, some users experienced problems sending and loading messages in Slack. Others may have completed these actions successfully but more slowly than expected.
Earlier today, we deployed a routine update to part of our network infrastructure. During this work, we deprovisioned some of our proxy servers and provisioned a new set. Some of our webapp servers took longer than expected to sync with the new list of proxy servers, resulting in a spike in connection errors. This coincided with the regular 9:00 AM PST increase in traffic to our servers, which may have exacerbated impact.
By around 9:36 AM PST, error rates had decreased and user impact had completely subsided, most likely due to our automatic scaling functionality. However, we're continuing to analyze data from this event to fully understand the root cause and prevent similar issues in the future.
</p>
https://slack-status.com/2024-02/b4d41d5daac6431c
2024-02-25T20:27:04-08:002024-02-26T06:30:45-08:00Incident: Issues with the billing history page
Issue Summary:
From 2:58 PM PST on February 22, 2024, until 9:47 PM PST on February 25, 2024, some users were unable to access their workspace billing history.
We traced the issue to a recent backend code change and reverted the change, which fixed the issue for all affected users.
Thank you very much for your patience while we resolved this.
<p>
<small>8:27pm PST</small>
Customers may be unable to access their billing history at the moment. We are working to resolve this and will be back with an update in 30 minutes or as soon as we have news to share.
If you need urgent access to your billing history, please contact us at feedback@slack.com.
Thank you for bearing with us.
</p>
<p>
<small>8:57pm PST</small>
We have identified the cause of the problem and we're preparing a fix.
We'll provide another update in 30 minutes or as soon as the fix goes live, whichever comes first.
</p>
<p>
<small>9:27pm PST</small>
We're still working to get the billing history page back to normal, and we’ll continue to provide updates every 30 minutes.
</p>
<p>
<small>9:55pm PST</small>
We've resolved the issue and the billing history page should load normally once more. Please refresh your browser tab to fetch the fix.
We apologize for the disruption to your work.
</p>
<p>
<small>6:30am PST</small>
Issue Summary:
From 2:58 PM PST on February 22, 2024, until 9:47 PM PST on February 25, 2024, some users were unable to access their workspace billing history.
We traced the issue to a recent backend code change and reverted the change, which fixed the issue for all affected users.
Thank you very much for your patience while we resolved this.
</p>
https://slack-status.com/2024-02/548b72776056c03b
2024-02-22T15:10:05-08:002024-02-22T19:05:59-08:00Incident: Threads aren't loading for some users
Issue summary:
From 2:54 PM to 2:59 PM PDT on February 22, 2024, some users were having issues loading Threads.
We carried out a thorough investigation and observed trends, but found no evidence of an issue on our side. By around 2:59 PM PDT, error rates had already subsided and we received confirmation from several users that they could load Threads again by reloading Slack.
Whilst no remedial action was taken on our end, we're analyzing data from this event to understand ways to mitigate similar issues that may occur in the future.
We're confident that there will be no further impact to users.
<p>
<small>3:10pm PST</small>
We are currently investigating an issue which is preventing users from being able to view threads. We apologize for the disruption, and we'll be back with an update as soon as we know more.
</p>
<p>
<small>3:27pm PST</small>
We're seeing signs of improvement but we're still monitoring the situation. Some users have reported that refreshing Slack via Ctrl + R (Windows/Linux) or Cmd + R (macOS) fixes the issue. We’ll be back with another update as soon as the issue is fully resolved.
</p>
<p>
<small>3:42pm PST</small>
Users should no longer be experiencing any issues with viewing threads. Thank you for your patience while we sorted this out.
</p>
<p>
<small>7:05pm PST</small>
Issue summary:
From 2:54 PM to 2:59 PM PDT on February 22, 2024, some users were having issues loading Threads.
We carried out a thorough investigation and observed trends, but found no evidence of an issue on our side. By around 2:59 PM PDT, error rates had already subsided and we received confirmation from several users that they could load Threads again by reloading Slack.
Whilst no remedial action was taken on our end, we're analyzing data from this event to understand ways to mitigate similar issues that may occur in the future.
We're confident that there will be no further impact to users.
</p>
https://slack-status.com/2024-02/83f4c18fc4d11c93
2024-02-21T19:19:25-08:002024-02-22T17:46:26-08:00Incident: Screen readers are not announcing unread messages when using the Ctrl + K command on Windows.
Issue summary:
On February 21, 2024 from 6:30 PM PST to 8:00 PM PST, customers using the JAWS or NVDA screen readers on Windows may have experienced issues with announcements for unread messages in the Quick Switcher (Ctrl + K).
A recent code change inadvertently introduced a conflict with an HTML label for an interactive element.
We reverted the code change, fixing the issue and restoring normal announcement behaviour.
<p>
<small>7:19pm PST</small>
Some customers who use screen readers on Windows are reporting they can no longer hear their unread messages when they use the Ctrl + K command. Our team are investigating the root cause of this issue. We’ll be back with another update as soon as we have more information to share.
</p>
<p>
<small>9:40am PST</small>
We've rolled out a fix and screen readers are now working as expected. If you're still encountering any trouble, please reload Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux). Thank you for your patience while we sorted out this issue.
</p>
<p>
<small>5:46pm PST</small>
Issue summary:
On February 21, 2024 from 6:30 PM PST to 8:00 PM PST, customers using the JAWS or NVDA screen readers on Windows may have experienced issues with announcements for unread messages in the Quick Switcher (Ctrl + K).
A recent code change inadvertently introduced a conflict with an HTML label for an interactive element.
We reverted the code change, fixing the issue and restoring normal announcement behaviour.
</p>
https://slack-status.com/2024-02/e04276624660caf9
2024-02-21T23:46:24-08:002024-02-22T06:32:08-08:00Incident: Trouble with password resets
Issue Summary:
Between 1:18 AM PST on February 21, 2024, and 5:35 AM PST on February 22, 2024, some users experienced issues resetting their passwords, especially when trying to set up two-factor authentication.
We traced the issue to a recent backend code change and reverted the change, which fixed the issue for all affected users.
Thank you for your patience while we resolved this and we apologize for any disruption to your work day.
<p>
<small>11:46pm PST</small>
Some users may be experiencing issues resetting their passwords, particularly when trying to set up two-factor authentication. We're investigating and we're very sorry for the disruption.
</p>
<p>
<small>5:55am PST</small>
Users should no longer be experiencing any trouble with password resets. Thank you very much for your patience while we worked on getting this fixed.
</p>
<p>
<small>6:32am PST</small>
Issue Summary:
Between 1:18 AM PST on February 21, 2024, and 5:35 AM PST on February 22, 2024, some users experienced issues resetting their passwords, especially when trying to set up two-factor authentication.
We traced the issue to a recent backend code change and reverted the change, which fixed the issue for all affected users.
Thank you for your patience while we resolved this and we apologize for any disruption to your work day.
</p>
https://slack-status.com/2024-01/f39851209d6c471a
2024-01-23T13:28:00-08:002024-02-22T06:31:13-08:00Incident: Some users are unable to upload, download, and view files in Slack.
Issue Summary:
Between 1:18 AM PST on February 21, 2024, and 5:35 AM PST on February 22, 2024, some users experienced issues resetting their passwords, especially when trying to set up two-factor authentication.
We traced the issue to a recent backend code change and reverted the change, which fixed the issue for all affected users.
Thank you for your patience while we resolved this and we apologize for any disruption to your work day.
<p>
<small>1:28pm PST</small>
Users may be having trouble viewing images as they are not loading, while others may be able to complete these actions but more slowly than expected.
We're actively looking into this, and we'll report back as more information becomes available.
</p>
<p>
<small>2:04pm PST</small>
We're seeing signs of recovery but uploading/downloading remain degraded though some attempts are successful. Our investigation is ongoing and we'll be back with another update as soon as information is available.
</p>
<p>
<small>2:42pm PST</small>
We are still seeing signs of improvement but are closely monitoring the issue as we work to identify the cause. We'll be back with another update once we have more news to share.
</p>
<p>
<small>5:28pm PST</small>
The issue that some users experienced when uploading, downloading, and viewing files in Slack should no longer be affecting users. We will share a summary of the issue that will include additional details and findings once available. We apologize for any disruptions to your day.
</p>
<p>
<small>4:22pm PST</small>
Issue Summary:
From 1:00pm PST on January 23, 2024 to 2:30pm PST on January 23, 2024, some users encountered trouble uploading, downloading, and viewing files in Slack.
We determined that an API call was not functioning correctly, and made a change to mitigate the issue. The upload problems were caused by a slowing down of responses due to a higher than normal volume of API calls due to the malfunction. This issue has been resolved and steps have been taken to avoid it happening in the future.
</p>
<p>
<small>6:31am PST</small>
Issue Summary:
Between 1:18 AM PST on February 21, 2024, and 5:35 AM PST on February 22, 2024, some users experienced issues resetting their passwords, especially when trying to set up two-factor authentication.
We traced the issue to a recent backend code change and reverted the change, which fixed the issue for all affected users.
Thank you for your patience while we resolved this and we apologize for any disruption to your work day.
</p>
https://slack-status.com/2024-02/16aecfd53af7219b
2024-02-08T13:24:54-08:002024-02-15T06:40:11-08:00Incident: Profile photos aren't displaying properly
Issue summary:
Starting around 12:46 PM PST on February 6, 2024, until around 4:06 PM PST on February 9, 2024, some users experienced issues with their profile pictures loading, resulting in the default avatar image displaying instead of their custom image.
We identified a bug that caused missing image data from a database cache. We fixed the bug and reset the backend cache to pull new data, which initially fixed the issue around 4:00 PM PST on February 6. However, some users began experiencing the problem again once their Slack client refreshed.
We continued investigating the issue, tracing it back to a piece of code that would push older image data to a database that housed profile information, and promptly reverted the incorrect code which resolved the issue for all affected users.
<p>
<small>1:24pm PST</small>
We've been investigating an ongoing issue where profile photos aren't displaying properly for some of our users. In order to facilitate improved communication and transparency, we'll be sharing any new updates here.
The cause of the issue has been identified and we're working on getting a fix out as soon as possible. We'll be back once we have another update to share.
</p>
<p>
<small>12:37pm PST</small>
A small subset of users may still experience trouble with their profile pictures displaying, though the majority of profile photos should be restored after a hard refresh of their Slack desktop client or after clearing app cache for mobile clients.
To conduct a hard refresh on your desktop app, you can use Cmd/Ctrl + Shift + R. For affected users on mobile, you may need to follow steps specific to your device to clear your Slack app cache.
We continue to work toward fully resolving this and will be back with another update once we have news to share.
</p>
<p>
<small>8:47am PST</small>
The issue some users experienced with profile pictures not displaying properly has been fixed. If you're still encountering the issue, a hard refresh of your Slack client (Cmd/Ctrl + Shift + R) is recommended. For mobile users, you may need to clear your Slack app cache.
We appreciate your patience while we sorted this out and apologize for the inconvenience. We'll follow up with a summary of the issue once it's available to share.
</p>
<p>
<small>6:40am PST</small>
Issue summary:
Starting around 12:46 PM PST on February 6, 2024, until around 4:06 PM PST on February 9, 2024, some users experienced issues with their profile pictures loading, resulting in the default avatar image displaying instead of their custom image.
We identified a bug that caused missing image data from a database cache. We fixed the bug and reset the backend cache to pull new data, which initially fixed the issue around 4:00 PM PST on February 6. However, some users began experiencing the problem again once their Slack client refreshed.
We continued investigating the issue, tracing it back to a piece of code that would push older image data to a database that housed profile information, and promptly reverted the incorrect code which resolved the issue for all affected users.
</p>
https://slack-status.com/2024-02/804d3a52e234f2fa
2024-02-12T08:25:49-08:002024-02-13T14:15:24-08:00Incident: Some users may be experiencing trouble with canvas.
Issue summary:
On February 12, 2024 from 6:00 AM PST to 8:32 AM PST, some users experienced latency when loading canvases or errors related to editing and saving canvases.
Upon investigation we determined that this was caused by rate limits put in place by an upstream provider. We restarted the impacted endpoints to mitigate the issue immediately, which resolved the issue for all affected users. We're continuing to partner with our upstream provider to reduce the likelihood of this occurring in the future.
<p>
<small>8:25am PST</small>
A small number of users may be experiencing trouble loading, editing or saving changes made to canvas.
We’re investigating and will let you know as soon as we know more.
We appreciate your patience in the meantime.
</p>
<p>
<small>9:17am PST</small>
Users should no longer be experiencing any trouble using canvas.
Apologies for the trouble today and thank you for your patience.
</p>
<p>
<small>2:15pm PST</small>
Issue summary:
On February 12, 2024 from 6:00 AM PST to 8:32 AM PST, some users experienced latency when loading canvases or errors related to editing and saving canvases.
Upon investigation we determined that this was caused by rate limits put in place by an upstream provider. We restarted the impacted endpoints to mitigate the issue immediately, which resolved the issue for all affected users. We're continuing to partner with our upstream provider to reduce the likelihood of this occurring in the future.
</p>
https://slack-status.com/2024-02/558e3bb8ce654659
2024-02-07T09:52:19-08:002024-02-07T18:06:20-08:00Incident: Some users may be having trouble connecting to Slack.
Issue summary:
On February 7, 2024 from around 9:23 AM PST to 9:51 AM PST, a small percentage of users may have experienced issues connecting to Slack or searching within Slack.
Our automated alerting systems detected an unusual spike in traffic to one of our servers. We carried out a thorough investigation, but the load subsided naturally as we worked, resolving the issue for all affected users.
While we did not take any remedial action in this case, we continued our investigation to better understand the problem and potential mitigation strategies for similar issues in future.
<p>
<small>9:52am PST</small>
A small percentage of users may be experiencing trouble connecting to Slack.
We’re investigating and will let you know as soon as we know more. We appreciate your patience in the meantime.
</p>
<p>
<small>10:25am PST</small>
We’re seeing signs of improvement, but we’re still monitoring the situation. We'll be back with another update once we have more news to share.
</p>
<p>
<small>11:24am PST</small>
Some users may no longer have trouble connecting to Slack, but we are continuing to monitor the situation closely and investigate the root cause.
We appreciate your patience while we sort this out.
</p>
<p>
<small>1:14pm PST</small>
Users should no longer be experiencing any trouble connecting to Slack. We apologize for the disruption and thank you for your patience.
</p>
<p>
<small>6:06pm PST</small>
Issue summary:
On February 7, 2024 from around 9:23 AM PST to 9:51 AM PST, a small percentage of users may have experienced issues connecting to Slack or searching within Slack.
Our automated alerting systems detected an unusual spike in traffic to one of our servers. We carried out a thorough investigation, but the load subsided naturally as we worked, resolving the issue for all affected users.
While we did not take any remedial action in this case, we continued our investigation to better understand the problem and potential mitigation strategies for similar issues in future.
</p>
https://slack-status.com/2024-02/30dfe547672802a4
2024-02-01T01:53:46-08:002024-02-01T06:43:18-08:00Incident: Users in Germany having trouble receiving 2FA SMS codes
Issue Summary:
On February 1, 2024 between 12:57 AM PST and 3:52 AM PST, some users in Germany were having trouble receiving two-factor authentication codes via SMS.
This was caused by an issue with a service provider and has now been resolved.
Users in Germany should no longer be having trouble and may also choose to use an authentication app to receive their 2FA codes instead of SMS.
Thank you for being patient with us, we appreciate it.
<p>
<small>1:53am PST</small>
We’re investigating an issue impacting users in Germany who may not be able to receive two-factor authentication SMS codes. We’ll report back with an update as soon as we have more information to share.
</p>
<p>
<small>4:25am PST</small>
Customers with German numbers should no longer be experiencing any trouble requesting two-factor authentication codes via SMS. If you have trouble setting up SMS 2FA you can also use an authentication app to set up 2FA. Please send a note to feedback@slack.com if you continue to have any trouble.
</p>
<p>
<small>6:43am PST</small>
Issue Summary:
On February 1, 2024 between 12:57 AM PST and 3:52 AM PST, some users in Germany were having trouble receiving two-factor authentication codes via SMS.
This was caused by an issue with a service provider and has now been resolved.
Users in Germany should no longer be having trouble and may also choose to use an authentication app to receive their 2FA codes instead of SMS.
Thank you for being patient with us, we appreciate it.
</p>
https://slack-status.com/2024-01/8e9a7cb95549d34c
2024-01-31T08:01:15-08:002024-01-31T10:14:04-08:00Incident: Users may be experiencing issues loading threads and sending mesages.
Issue Summary:
On January 31, 2024 between 7:43 AM PST and 8:27 AM PST, some users were unable to load threads and send messages.
We traced the issue to a backend failure and immediately implemented a change which fixed the issue for all affected users.
We apologize for any disruption to your work day and appreciate your patience while we resolved the issue.
<p>
<small>8:01am PST</small>
Some users may be experiencing issues loading threads.
We're actively looking into this, and we'll report back as more information becomes available.
We appreciate your patience in the meantime.
</p>
<p>
<small>8:50am PST</small>
We've resolved the issue with threads not loading. Through our investigation, we determined that some users may have also experienced issues sending messages or loading channel content, users should no longer be experiencing trouble.
Thank you very much for your patience while we worked on getting this fix out the door.
</p>
<p>
<small>10:14am PST</small>
Issue Summary:
On January 31, 2024 between 7:43 AM PST and 8:27 AM PST, some users were unable to load threads and send messages.
We traced the issue to a backend failure and immediately implemented a change which fixed the issue for all affected users.
We apologize for any disruption to your work day and appreciate your patience while we resolved the issue.
</p>
https://slack-status.com/2024-01/0a000b0ad09623a0
2024-01-24T12:00:49-08:002024-01-30T09:58:00-08:00Incident: Some latency with Slack
Issue summary:
From 11:37 AM PST to 11:48 AM PST on January 24, 2024, we experienced an unexpected amount of API calls to our servers that occurred within a short window of time. The volume of API calls resulted in some users experiencing latency loading channels and general difficulties connecting to Slack. We investigated the impact with a broad lens and began to observe signs of recovery around 12:09 PM PST.
During this time, we cautiously observed our health metrics to ensure our servers were accurately recovering. As a result of our thorough monitoring, there was a delay before we could confirm the issue was fully resolved. Our teams have also put in measures to help reduce the likelihood of this occurring again in the future.
<p>
<small>12:00pm PST</small>
Users are experiencing general delays with the service, including issues with connecting to Slack and loading channels. We're looking into this and will share an update once we have more information.
</p>
<p>
<small>12:28pm PST</small>
Our investigation is still ongoing at this time. We'll report back once we have news to share. Thank you for your patience.
</p>
<p>
<small>12:37pm PST</small>
We're seeing signs of recovery, but are still investigating the root cause of the issue. We'll be back with an update as soon as we have more info to share.
</p>
<p>
<small>5:27pm PST</small>
The issue that was causing some users to experience latency with connecting to Slack and loading channels has been resolved.
We apologize for any disruptions to your day.
</p>
<p>
<small>9:58am PST</small>
Issue summary:
From 11:37 AM PST to 11:48 AM PST on January 24, 2024, we experienced an unexpected amount of API calls to our servers that occurred within a short window of time. The volume of API calls resulted in some users experiencing latency loading channels and general difficulties connecting to Slack. We investigated the impact with a broad lens and began to observe signs of recovery around 12:09 PM PST.
During this time, we cautiously observed our health metrics to ensure our servers were accurately recovering. As a result of our thorough monitoring, there was a delay before we could confirm the issue was fully resolved. Our teams have also put in measures to help reduce the likelihood of this occurring again in the future.
</p>
https://slack-status.com/2024-01/8ad463b92a084387
2024-01-29T15:38:14-08:002024-01-29T19:44:32-08:00Incident: Pasting via Cmd/Ctrl + V not working
Issue summary:
From 11:45 AM PST to around 5:20 PM PST on January 29, 2024, some customers experienced problems using keyboard shortcuts to paste text into Slack.
A code change inadvertently introduced an issue that prevented the use of Cmd/Ctrl + V to paste text into Slack. We reverted this change, then deployed a fix to fully resolve the issue.
<p>
<small>3:38pm PST</small>
We are currently investigating an issue which is preventing users from pasting via Ctrl + V (Windows/Linux) and Cmd + V (MacOS). We'll be back with another updates as soon as we know more.
In the meantime, users can workaround this issue by pasting via mouse (right click to open context menu + select Paste).
</p>
<p>
<small>6:40pm PST</small>
We have resolved the issue, and you should be able to paste into Slack with keyboard shortcuts as normal again.
If you're still experiencing problems with pasting, please refresh Slack with Cmd/Ctrl + Shift + R.
Thank you for bearing with us, we appreciate it.
</p>
<p>
<small>7:44pm PST</small>
Issue summary:
From 11:45 AM PST to around 5:20 PM PST on January 29, 2024, some customers experienced problems using keyboard shortcuts to paste text into Slack.
A code change inadvertently introduced an issue that prevented the use of Cmd/Ctrl + V to paste text into Slack. We reverted this change, then deployed a fix to fully resolve the issue.
</p>
https://slack-status.com/2024-01/8b623c0b5640c28f
2024-01-24T08:37:35-08:002024-01-26T06:18:08-08:00Incident: Service wide connection issues
Issue summary:
On January 24, 2024 from 5:40 AM PST to around 9:00 AM PST, a small number of users experienced issues connecting to Slack and running workflows.
A change to routing in our servers resulted in requests failing due to a lack of available resources. This change was rolled back and Slack returned to a normal state.
<p>
<small>8:37am PST</small>
Some customers are encountering issues connecting to Slack and running workflows. We're actively looking into this, and we'll report back as more information becomes available.
[This post has been updated to correct a grammatical error.]
</p>
<p>
<small>9:05am PST</small>
We’ve isolated the problem and we’re focused on getting Slack working again as soon as possible. Thank you for your continued patience.
</p>
<p>
<small>9:19am PST</small>
We’re seeing signs of improvement, but we’re still monitoring the situation. We’ll confirm once the issue is fully resolved or we have another update to share.
</p>
<p>
<small>9:43am PST</small>
We’ve implemented a fix and all impacted features are up and running once again. We appreciate your patience while we sorted this out. If you’re still encountering any trouble, please reload Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux). Thank you for your patience while we sorted out this issue.
</p>
<p>
<small>6:18am PST</small>
Issue summary:
On January 24, 2024 from 5:40 AM PST to around 9:00 AM PST, a small number of users experienced issues connecting to Slack and running workflows.
A change to routing in our servers resulted in requests failing due to a lack of available resources. This change was rolled back and Slack returned to a normal state.
</p>
https://slack-status.com/2024-01/166eb312bd134031
2024-01-22T13:15:21-08:002024-01-22T19:06:24-08:00Incident: Some Enterprise Grid users may be seeing unexpected Slackbot responses
Issue summary:
From 12:20pm PST until 3:00pm PST on January 22, 2024, some Enterprise Grid users noticed multiple Slackbot responses being triggered unexpectedly.
We determined that the rollout of a fix for an older bug report pertaining to Slackbot responses not working in org-wide or multi-workspace channels, was the root cause.
Whilst the fix for the bug was intended to improve this features behaviour, ensuring that Slackbot custom responses would work in these channel types, our wider team concluded that the fixed behaviour might not function well for organizations with potentially thousands of custom Slackbot responses.
We rolled back the deployment which caused this behaviour. Customers will no longer see these Slackbot custom responses being triggered unexpectedly.
A discussion is underway about the long-term future of Slackbot custom response behaviour within large organizations.
Thank you for your patience whilst we resolved this.
<p>
<small>1:15pm PST</small>
Some Enterprise Grid users may notice multiple Slackbot responses being triggered unexpectedly. We are aware of the issue and are investigating.
</p>
<p>
<small>1:47pm PST</small>
We have identified the code that was causing the issue and have rolled it back. We are currently working on a more permanent fix to ensure this does not happen again.
</p>
<p>
<small>2:29pm PST</small>
To clarify our last update, we have started the process of rolling back the code, but that work is not yet complete, and some users may still be affected. We apologize for this miscommunication.
We believe this revert will resolve the issue and are actively working to complete this process as soon as we can.
</p>
<p>
<small>3:14pm PST</small>
We've completed rolling back the code which cause this issue, and users should no longer be seeing these Slackbot responses. Thank you for bearing with us on this one, and we apologize for the disruption to your day.
</p>
<p>
<small>7:06pm PST</small>
Issue summary:
From 12:20pm PST until 3:00pm PST on January 22, 2024, some Enterprise Grid users noticed multiple Slackbot responses being triggered unexpectedly.
We determined that the rollout of a fix for an older bug report pertaining to Slackbot responses not working in org-wide or multi-workspace channels, was the root cause.
Whilst the fix for the bug was intended to improve this features behaviour, ensuring that Slackbot custom responses would work in these channel types, our wider team concluded that the fixed behaviour might not function well for organizations with potentially thousands of custom Slackbot responses.
We rolled back the deployment which caused this behaviour. Customers will no longer see these Slackbot custom responses being triggered unexpectedly.
A discussion is underway about the long-term future of Slackbot custom response behaviour within large organizations.
Thank you for your patience whilst we resolved this.
</p>
https://slack-status.com/2024-01/9301a4fb7cc577ea
2024-01-15T07:36:47-08:002024-01-15T15:51:07-08:00Incident: Some users may not be able to configure 2FA
Issue summary:
From 11:26 AM PST on January 12, 2024 to 10:04 AM PST on January 15, 2024, some users were unable to configure 2FA on their accounts. We were made aware of this after a spike in reports early in the morning of Monday, January 15.
Upon investigation, this issue was traced back to a recent code change which we discovered was preventing users from being redirected back to the 2FA configuration page after entering their password during the 2FA setup process. We immediately reverted this change which fully resolved the issue.
<p>
<small>7:36am PST</small>
Some users may be having trouble setting up 2FA.
In the meantime, we recommend using an incognito tab or a private browsing to set up 2FA while we continue actively looking into this issue.
We apologize for the inconvenience and we'll report back as more information becomes available.
</p>
<p>
<small>10:01am PST</small>
We’ve identified the cause of this issue and we’re working on getting a fix out as soon as possible. We’ll be back once we have another update to share.
</p>
<p>
<small>10:26am PST</small>
We’ve resolved the issue with 2FA setup. Thank you very much for your patience while we worked on getting this fix out the door.
If you have any continued trouble, please reach out to feedback@slack.com
</p>
<p>
<small>3:51pm PST</small>
Issue summary:
From 11:26 AM PST on January 12, 2024 to 10:04 AM PST on January 15, 2024, some users were unable to configure 2FA on their accounts. We were made aware of this after a spike in reports early in the morning of Monday, January 15.
Upon investigation, this issue was traced back to a recent code change which we discovered was preventing users from being redirected back to the 2FA configuration page after entering their password during the 2FA setup process. We immediately reverted this change which fully resolved the issue.
</p>
https://slack-status.com/2023-12/7f05f584fa98cf7c
2023-12-19T15:56:51-08:002023-12-20T18:01:15-08:00Incident: Users seeing blurred file unfurls
Issue summary:
From 1:43 PM PST on December 19, 2023 to 10:25 AM PST on December 20, 2023, a small number of users experienced seeing blurred file previews. We discovered that this affected users with international data residency enabled in their workspace.
A change was made to our file storage, resulting in unexpected behaviour with file previews. We promptly identified the problematic code and reverted back the change in an effort to prevent file previews from becoming blurred.
We later ran a script to fix affected files to display with clear previews, resolving the issue for all affected users.
<p>
<small>3:56pm PST</small>
We're receiving reports about blurred images in Slack. The image does display as expected after clicking the image preview.
We're on the case and will provide another update once we have more details to share. We're sorry for the disruption.
</p>
<p>
<small>4:26pm PST</small>
Further to reports about blurred images in Slack, we're also receiving reports on issues with downloading and viewing PDF files.
We're currently reviewing the scope of impact to see what customers could be affected. We will provide further updates once we have more details to share. We're sorry for the disruption.
</p>
<p>
<small>5:05pm PST</small>
Customers have reported seeing blurred images within Slack until clicking the preview option. It's also been reported that PDF files are not downloading as the correct file type.
Our engineers are in the process of investigating the root cause and are working hard to resolve this for users.
We will provide further updates once our engineers have a resolution in progress. We're sorry for the disruption.
</p>
<p>
<small>5:37pm PST</small>
Customers have reported seeing blurred images within Slack until clicking the preview option. It's also been reported that PDF files are not downloading as the correct file type.
Our engineers are in the testing stage and are reverting some items in the back-end, which has lead to them seeing small improvements with file unfurl behaviour.
We will provide further updates once our engineers have a resolution in progress. We're sorry for the disruption.
</p>
<p>
<small>6:47pm PST</small>
We have identified the cause of images being blurry in Slack and customers can now upload and preview new images without issue.
We are still continuing to work on regenerating the previews of older affected images.
We will provide an update when functionality is fully restored. Thank you for bearing with us in the meantime.
</p>
<p>
<small>7:37pm PST</small>
We're continuing to work towards a fix and will provide an update when this is completely resolved.
In the meantime and as a workaround, if the image remains blurry, please expand the image and Slack will load the full size version.
</p>
<p>
<small>10:51am PST</small>
We've resolved the issue with blurred file previews in Slack. Previously uploaded files should no longer display a blurred preview.
It's recommended for anyone still seeing the issue to hard refresh their Slack app by pressing Cmd/Ctrl + Shift + R.
We apologize for the trouble this may have caused and appreciate your patience while we sorted things out.
</p>
<p>
<small>6:01pm PST</small>
Issue summary:
From 1:43 PM PST on December 19, 2023 to 10:25 AM PST on December 20, 2023, a small number of users experienced seeing blurred file previews. We discovered that this affected users with international data residency enabled in their workspace.
A change was made to our file storage, resulting in unexpected behaviour with file previews. We promptly identified the problematic code and reverted back the change in an effort to prevent file previews from becoming blurred.
We later ran a script to fix affected files to display with clear previews, resolving the issue for all affected users.
</p>
https://slack-status.com/2023-12/b0072b040087e20b
2023-12-03T16:56:55-08:002023-12-04T20:20:17-08:00Incident: Trouble loading some parts of Slack
Issue summary:
On December 3, 2023 from 4:35 PM PST to around 5:29 PM PST, a small number of customers experienced trouble loading parts of Slack such as threads, channels, and direct messages. Affected customers may have also had trouble sending messages and running some workflows.
One of our databases experienced a minor hardware issue. This triggered an automatic intervention from our redundancy systems, and a new database was brought into service. However, the new database entered a state where it could not successfully respond to incoming queries.
As an immediate mitigation step, we performed a manual replacement and redirected the incoming queries to a third database. This resolved the issue for all impacted customers.
<p>
<small>4:56pm PST</small>
Some customers may be experiencing issues loading the Threads view at the moment. We are actively investigating and we'll provide an update in 30 minutes.
We apologize for the disruption to your work.
</p>
<p>
<small>5:27pm PST</small>
In addition to the Threads view, customers may also be running in to trouble with blank channels and direct message conversations, issues with some workflows, and problems sending messages. We are working to resolve this as fast as we can. We'll be back with more information in 30 minutes.
Note: We've changed the incident title to better reflect the scope of the issues.
</p>
<p>
<small>5:58pm PST</small>
We have resolved the issue, and Slack should be back to full functionality for everyone. If you run in to any lingering issues, please refresh Slack with Cmd/Ctrl + Shift + R.
Thank you for bearing with us, we appreciate it.
</p>
<p>
<small>8:20pm PST</small>
Issue summary:
On December 3, 2023 from 4:35 PM PST to around 5:29 PM PST, a small number of customers experienced trouble loading parts of Slack such as threads, channels, and direct messages. Affected customers may have also had trouble sending messages and running some workflows.
One of our databases experienced a minor hardware issue. This triggered an automatic intervention from our redundancy systems, and a new database was brought into service. However, the new database entered a state where it could not successfully respond to incoming queries.
As an immediate mitigation step, we performed a manual replacement and redirected the incoming queries to a third database. This resolved the issue for all impacted customers.
</p>
https://slack-status.com/2023-11/edc9f768095e939b
2023-11-28T23:40:53-08:002023-11-29T06:41:20-08:00Incident: Slack not loading for some users
Issue Summary:
From 9:30 AM PST on November 28, 2023 to 12:30 AM PST on November 29, 2023, some users in Asia were experiencing issues with Slack loading. Most of the impacted users were in Japan and India.
We traced this back to an unexpected spike in a database load for which we quickly saw recovery.
We monitored the situation closely and as the load reduced without intervention, we did not take any mitigating actions.
<p>
<small>11:40pm PST</small>
We’re investigating the issue where Slack is not loading for some users. We’re looking into the cause and will provide more information as soon as it's available.
</p>
<p>
<small>12:48am PST</small>
This issue with Slack not loading for some users is now resolved. We apologize for any inconvenience caused by this.
</p>
<p>
<small>6:41am PST</small>
Issue Summary:
From 9:30 AM PST on November 28, 2023 to 12:30 AM PST on November 29, 2023, some users in Asia were experiencing issues with Slack loading. Most of the impacted users were in Japan and India.
We traced this back to an unexpected spike in a database load for which we quickly saw recovery.
We monitored the situation closely and as the load reduced without intervention, we did not take any mitigating actions.
</p>
https://slack-status.com/2023-11/2b97ec921a81988a
2023-11-08T08:30:57-08:002023-11-17T13:42:44-08:00Incident: Some users may be experiencing issues with huddles
Issue Summary:
On November 8, 2023 from 7:55 AM PT until 8:27 PM PT, a small number of users experienced errors when attempting to join or start huddles.
We traced this back to an unexpected spike in a database load for which we quickly saw recovery.
We monitored the situation closely and in an effort to prevent further unexpected error spikes, we implemented a strategic configuration adjustment to optimize the load on our database.
Affected users should no longer experience issues with huddles.
<p>
<small>8:30am PST</small>
Some users may be experiencing trouble with huddles.
We’re investigating and will let you know as soon as we know more. We appreciate your patience in the meantime.
</p>
<p>
<small>9:18am PST</small>
We're continuing to work towards a fix, and will provide updates here when new information becomes available.
We appreciate your patience in the meantime.
</p>
<p>
<small>12:19pm PST</small>
This issue is now resolved and users should no longer encounter issues in huddles. We apologize for any disruption this brought to your day.
</p>
<p>
<small>1:42pm PST</small>
Issue Summary:
On November 8, 2023 from 7:55 AM PT until 8:27 PM PT, a small number of users experienced errors when attempting to join or start huddles.
We traced this back to an unexpected spike in a database load for which we quickly saw recovery.
We monitored the situation closely and in an effort to prevent further unexpected error spikes, we implemented a strategic configuration adjustment to optimize the load on our database.
Affected users should no longer experience issues with huddles.
</p>
https://slack-status.com/2023-11/0e7bc0e7a7e7cd87
2023-11-14T15:28:30-08:002023-11-14T19:01:39-08:00Outage: Slack not loading for some users
Issue summary:
From 3:15 PM PST on November 14, 2023 to around 4:13 PM PST, a small number of customers using the Slack desktop app were unable to connect to Slack. This may have manifested as a "Something's gone awry" error page.
A recent code change inadvertently introduced a logic error that prevented the desktop app from connecting as expected. We reverted this code change as an immediate mitigation step, then rolled out a fix to correct the logic error.
The fix resolved the issue for all affected customers, restoring full access to Slack.
<p>
<small>3:28pm PST</small>
We're investigating reports of Slack not loading for some users. We'll share an update once we have more information, we appreciate your patience.
</p>
<p>
<small>3:58pm PST</small>
We believe we have identified the cause of this issue and are working on a fix. We'll be back with an update soon.
Please use Slack in a browser window in the meantime if possible.
</p>
<p>
<small>4:29pm PST</small>
We have begun deploying a fix and it's rolling out to everyone gradually.
We appreciate your patience as we roll this out and restore normal service for everyone.
</p>
<p>
<small>4:55pm PST</small>
We’ve completed rolling out a fix and Slack is now working as expected for all users. If you’re still encountering any trouble, please reload Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux).
We’re sorry for the disruption we brought to your day.
</p>
<p>
<small>7:01pm PST</small>
Issue summary:
From 3:15 PM PST on November 14, 2023 to around 4:13 PM PST, a small number of customers using the Slack desktop app were unable to connect to Slack. This may have manifested as a "Something's gone awry" error page.
A recent code change inadvertently introduced a logic error that prevented the desktop app from connecting as expected. We reverted this code change as an immediate mitigation step, then rolled out a fix to correct the logic error.
The fix resolved the issue for all affected customers, restoring full access to Slack.
</p>
https://slack-status.com/2023-11/16fed44d7948cf49
2023-11-09T12:20:26-08:002023-11-13T07:57:36-08:00Incident: User presence is unexpectedly changing
Issue Summary:
From 4:00 PM PDT on October 31, 2023 to 2:00 PM PDT on November 9, 2023, user presence would unexpectedly change to away or inactive.
We determined that an error during a routine system optimization on connectivity state caused this issue.
We reverted the change which fixed the issue for all affected customers.
Thank you for your patience while we resolved this.
<p>
<small>12:20pm PST</small>
We're investigating an issue where user presence is unexpectedly changing to away or active. We are also working to understand if this is a continuation of the issue we investigated yesterday. We'll be back with an update once more information is available. Thank you for your patience.
</p>
<p>
<small>1:02pm PST</small>
We’re still actively investigating the issue with user presence, but we don’t have any new information to share at this time. Thanks for sticking with us as we continue to work towards a fix.
</p>
<p>
<small>11:12am PST</small>
We're seeing signs of improvement but are continuing to monitor the situation. We'll provide another update once we have more information to share.
</p>
<p>
<small>3:12pm PST</small>
We resolved the issue with user presence unexpectedly changing for some users. Affected users may need to complete a hard refresh of their Slack app (Cmd/Ctrl + Shift + R) to capture the fix.
We apologize for any inconvenience this has caused and appreciate your patience while we sorted this out.
</p>
<p>
<small>7:57am PST</small>
Issue Summary:
From 4:00 PM PDT on October 31, 2023 to 2:00 PM PDT on November 9, 2023, user presence would unexpectedly change to away or inactive.
We determined that an error during a routine system optimization on connectivity state caused this issue.
We reverted the change which fixed the issue for all affected customers.
Thank you for your patience while we resolved this.
</p>
https://slack-status.com/2023-11/ea3a2a3e32e79902
2023-11-08T10:47:14-08:002023-11-08T20:29:31-08:00Incident: Issues with user status, read state, and file previews
Issue summary:
From 10:00 PM PST on November 7, 2023 to 1:15 PM PST on November 8, 2023, some users experienced issues with their user status not updating, removing previews, and being unable to mark channels as read. Some keyboard shortcuts were also affected and were unable to be used.
We determined that a recent code change was the root cause for the unexpected behaviour with these features.
To restore functionality, we reverted the related code. We then did some additional testing and monitoring to confirm all issues were fully resolved and Slack was operating as expected.
<p>
<small>10:47am PST</small>
We’re looking into reports of user status not updating, users being unable to remove previews, and marking channels as read. We’ll let you know as we learn more. Thank you for bearing with us in the meantime.
</p>
<p>
<small>11:17am PST</small>
No additional news to share just yet, but we're focused on getting things back to normal as quickly as we can. We apologize for the continued trouble.
</p>
<p>
<small>11:45am PST</small>
We've discovered that this issue also impacts keyboard shortcuts such as "Cmd/Ctrl + K" or "Cmd/Ctrl + F". The root cause of this issue is not yet clear, but we're continuing to investigate this as a top priority. We'll be back with another update as soon as we have any new information to share. Thank you for continuing to bear with us.
</p>
<p>
<small>12:21pm PST</small>
No additional news to share just yet, but we’re focused on getting things back to normal as quickly as we can. We apologize for the continued trouble.
</p>
<p>
<small>12:44pm PST</small>
We've isolated the issue with keyboard shortcuts, marking items as read, and dismissing app previews. We are working to restore functionality, but continue investigating the root cause for user status not updating.
</p>
<p>
<small>1:48pm PST</small>
Thank you for your patience! We have made a change that restored functionality to keyboard shortcuts, marking items as read, and dismissing app previews. Users should no longer have issues with those features but may need to refresh their app or browser.
We are still investigating the issues related to user statuses and will share updates as new information becomes available.
</p>
<p>
<small>3:50pm PST</small>
We have made another change that has restored functionality to user statuses. Users should refresh if they do not see this change, but all issues should now be resolved.
</p>
<p>
<small>8:29pm PST</small>
Issue summary:
From 10:00 PM PST on November 7, 2023 to 1:15 PM PST on November 8, 2023, some users experienced issues with their user status not updating, removing previews, and being unable to mark channels as read. Some keyboard shortcuts were also affected and were unable to be used.
We determined that a recent code change was the root cause for the unexpected behaviour with these features.
To restore functionality, we reverted the related code. We then did some additional testing and monitoring to confirm all issues were fully resolved and Slack was operating as expected.
</p>
https://slack-status.com/2023-11/ef3e4b0ebcf16d8d
2023-11-03T17:36:09-07:002023-11-07T17:59:22-08:00Outage: Users unable to connect to Slack or send messages
Issue summary:
On November 4, 2023, between 5:09 PM PDT and 5:20 PM PDT, many customers were unable send messages or to connect to Slack.
A routine code change introduced a database error that prevented cached data from being cleared correctly, resulting in severe performance issues.
We rolled back the code change and refreshed all affected servers, resolving the issue for all users.
<p>
<small>5:36pm PDT</small>
Some users have been experiencing issues with being unable to connect to Slack or send messages. We are seeing a spike in connectivity errors and our team is currently investigating. We'll be back with another update as soon as we have more info to share.
</p>
<p>
<small>6:27pm PDT</small>
We resolved the issue users experienced when trying to connect to Slack or send messages. We appreciate your patience while we sorted this out and apologize for any disruption to your day.
</p>
<p>
<small>5:59pm PST</small>
Issue summary:
On November 4, 2023, between 5:09 PM PDT and 5:20 PM PDT, many customers were unable send messages or to connect to Slack.
A routine code change introduced a database error that prevented cached data from being cleared correctly, resulting in severe performance issues.
We rolled back the code change and refreshed all affected servers, resolving the issue for all users.
</p>
https://slack-status.com/2023-10/8c886b57284762b6
2023-10-31T17:22:32-07:002023-11-01T20:45:50-07:00Outage: Issues for customers enrolled in the French data region
Issue summary:
From 5:01 PM PDT on October 31, 2023 to around 5:54 PM PDT, customers with international data residency in the Paris, France region, experienced issues connecting to Slack and sending messages.
A routine credential rotation caused database sync issues for the Paris, France data residency region. We reverted this code change and restarted the affected databases, resolving the issue for all impacted customers.
Once we had mitigated the immediate impact and restored connectivity for customers in the Paris, France data residency region, we reviewed the credential rotation for all other data residency regions to ensure the same issue would not occur anywhere else.
Please note that the start and end time of the incident have been edited for accuracy.
<p>
<small>5:22pm PDT</small>
Some customers with international data residency in the Paris, France region may be experiencing issues connecting to Slack or sending messages. We're investigating and we'll be back with more information in 30 minutes or as soon as we have news to share.
We apologize for the disruption to your work. Thank you for bearing with us.
</p>
<p>
<small>5:57pm PDT</small>
We are investigating some recent code changes that may be causing issues, and we're preparing to roll these back. We are also working to prevent the impact from spreading to additional data regions.
We'll be back with an update in 30 minutes.
</p>
<p>
<small>6:22pm PDT</small>
We've pushed a potential fix and we're seeing significant improvements, though we're still monitoring the situation closely.
Affected customers may need to refresh Slack to reconnect. Please open Slack and press Cmd/Ctrl + Shift + R. Slack will reload but will not sign you out.
We'll share another update in 30 minutes or as soon as we have more information.
</p>
<p>
<small>6:53pm PDT</small>
We're continuing to monitor the situation and we'll provide an update in 30 minutes or as soon as connectivity is fully restored, whichever is soonest.
</p>
<p>
<small>7:24pm PDT</small>
We're completing some last checks to ensure that this issue is fully resolved for all customers.
We'll be back with an update as soon as we have confirmation that Slack is up and running for everyone.
</p>
<p>
<small>7:43pm PDT</small>
We have resolved the connectivity issue for all affected customers. Everyone should be able to connect as normal once again.
Thank you for your patience while we addressed this.
</p>
<p>
<small>8:45pm PDT</small>
Issue summary:
From 5:01 PM PDT on October 31, 2023 to around 5:54 PM PDT, customers with international data residency in the Paris, France region, experienced issues connecting to Slack and sending messages.
A routine credential rotation caused database sync issues for the Paris, France data residency region. We reverted this code change and restarted the affected databases, resolving the issue for all impacted customers.
Once we had mitigated the immediate impact and restored connectivity for customers in the Paris, France data residency region, we reviewed the credential rotation for all other data residency regions to ensure the same issue would not occur anywhere else.
Please note that the start and end time of the incident have been edited for accuracy.
</p>
https://slack-status.com/2023-10/2ef86432e31615ea
2023-10-24T00:38:17-07:002023-10-24T14:08:02-07:00Incident: A small number of users are having problems loading Slack.
Customers should no longer be experiencing any connection issues with Slack. Apologies for the trouble today and thank you for your patience.
<p>
<small>12:38am PDT</small>
A small number of users on desktop & web browsers are experiencing issues loading Slack. We’re actively investigating the issue and will report back as soon as we have more information to share. We apologise for the inconvenience.
</p>
<p>
<small>1:03am PDT</small>
Our investigation has revealed that this problem is affecting our fallback content delivery network. If you experience problems loading Slack, please try restarting your Slack app to restore functionality. We’ll let you know as soon as we have another update to share.
</p>
<p>
<small>7:05am PDT</small>
While we are working towards finding the cause of the issue, the problem with our fallback content delivery network appears to be affecting less than 1% of users. If you experience problems loading Slack, please restart your Slack app (ctrl/cmd+shift+R) to restore functionality. We’ll let you know as soon as we have more to share.
</p>
<p>
<small>2:08pm PDT</small>
Customers should no longer be experiencing any connection issues with Slack. Apologies for the trouble today and thank you for your patience.
</p>
https://slack-status.com/2023-10/ad8f0e62516e8812
2023-10-06T02:11:22-07:002023-10-06T08:28:47-07:00Outage: Users are having trouble connecting to Slack
Issue summary:
On Friday, October 6 2023, from 1:52 AM PDT to 2:12 AM PDT, some users were unable to load Slack. This was caused by an issue where certain backend Slack processes were pulling data from our database rather than their cache which caused strain on our servers.
We rolled out a change to how these processes load data, and the issue was resolved for all users.
<p>
<small>2:11am PDT</small>
Some customers are encountering various errors in Slack. We're actively looking into this, and we'll report back as more information becomes available.
</p>
<p>
<small>2:41am PDT</small>
We are seeing things get back to normal but are continuing to investigate and monitor the situation. We’ll be back with another update in 30 minutes.
</p>
<p>
<small>3:15am PDT</small>
Users should be noticing signs of improvement. We’re still looking into the root cause of any connection issues experienced and we’ll provide new information as it’s available. We apologise for any trouble caused.
</p>
<p>
<small>3:49am PDT</small>
We’re continuing to investigate the root cause of the connection issues, we’ll provide an update once it’s available. We apologise for any inconvenience caused.
</p>
<p>
<small>4:06am PDT</small>
Customers should no longer be experiencing any connection issues with Slack. Apologies for the trouble today and thank you for your patience.
</p>
<p>
<small>8:28am PDT</small>
Issue summary:
On Friday, October 6 2023, from 1:52 AM PDT to 2:12 AM PDT, some users were unable to load Slack. This was caused by an issue where certain backend Slack processes were pulling data from our database rather than their cache which caused strain on our servers.
We rolled out a change to how these processes load data, and the issue was resolved for all users.
</p>
https://slack-status.com/2023-10/6d592537eb7bd98e
2023-10-05T00:08:08-07:002023-10-05T06:53:12-07:00Incident: Trouble connecting to Slack
Issue Summary:
On Oct 4, 2023 from 11:15 PM PDT to around 11:58 PM PDT a small subset of users experienced trouble connecting to Slack and sending and receiving messages.
A code change optimizing resource usage caused a core service to restart causing brief connectivity issues for a few users.
The service recovered automatically and we’ve put measures in place to prevent this from happening in the future.
<p>
<small>12:36am PDT</small>
This connection issue has now been resolved.
Users should now be able to connect to Slack without further disruption. Please reload your application or browser to see this resolution.
</p>
<p>
<small>12:08am PDT</small>
Some customers may be experiencing issues connecting to Slack at the moment.
We're on the case and working to resolve this as fast as we can. Thank you for bearing with us.
</p>
<p>
<small>6:53am PDT</small>
Issue Summary:
On Oct 4, 2023 from 11:15 PM PDT to around 11:58 PM PDT a small subset of users experienced trouble connecting to Slack and sending and receiving messages.
A code change optimizing resource usage caused a core service to restart causing brief connectivity issues for a few users.
The service recovered automatically and we’ve put measures in place to prevent this from happening in the future.
</p>
https://slack-status.com/2023-10/ee32d13012fb29d5
2023-10-03T16:37:40-07:002023-10-04T11:53:01-07:00Incident: Issues with loading some items on Slack
Issue Summary:
On October 3, 2023 from 4:00 PM PDT to 4:20 PM PDT, some customers had trouble loading threads, mentions, reactions and search results in Slack.
We determined that this was caused during the process of changing our database hosts on our backend. To mitigate the issue, we stopped the change which fixed the issue for affected customers.
We're sorry for any disruption this may have caused and thank you for your patience while we looked into this
<p>
<small>4:37pm PDT</small>
Some users may be having trouble loading threads, mentions and reactions, and search results. We are currently investigating.
</p>
<p>
<small>5:14pm PDT</small>
This issue is now resolved for all users. Thank you again for bearing with us in the meantime.
</p>
<p>
<small>11:52am PDT</small>
Issue Summary:
On October 3, 2023 from 4:00 PM PDT to 4:20 PM PDT, some customers had trouble loading threads, mentions, reactions and search results in Slack.
We determined that this was caused during the process of changing our database hosts on our backend. To mitigate the issue, we stopped the change which fixed the issue for affected customers.
We're sorry for any disruption this may have caused and thank you for your patience while we looked into this.
</p>
<p>
<small>11:53am PDT</small>
Issue Summary:
On October 3, 2023 from 4:00 PM PDT to 4:20 PM PDT, some customers had trouble loading threads, mentions, reactions and search results in Slack.
We determined that this was caused during the process of changing our database hosts on our backend. To mitigate the issue, we stopped the change which fixed the issue for affected customers.
We're sorry for any disruption this may have caused and thank you for your patience while we looked into this
</p>
https://slack-status.com/2023-10/101eb09455dcaa9a
2023-10-02T13:16:28-07:002023-10-02T13:16:28-07:00Incident: Threads were not loading for some users
Issue Summary:
On October 2, 2023 from 12:18PM PDT to 12:26PM PDT, some users noticed that threads were not loading correctly.
We identified a database issue on our backend that was causing the threads to not load. The issue was mitigated and users should no longer experience trouble loading threads.
We are investigating the cause of the database issue so we can ensure this does not happen again.
We apologize for any interruption to your work day!
<p>
<small>1:16pm PDT</small>
Issue Summary:
On October 2, 2023 from 12:18PM PDT to 12:26PM PDT, some users noticed that threads were not loading correctly.
We identified a database issue on our backend that was causing the threads to not load. The issue was mitigated and users should no longer experience trouble loading threads.
We are investigating the cause of the database issue so we can ensure this does not happen again.
We apologize for any interruption to your work day!
</p>
https://slack-status.com/2023-09/7e84b360042dd07b
2023-09-28T13:05:12-07:002023-09-29T15:55:29-07:00Incident: Issues with receiving 2FA code
Issue summary:
On September 28, 2023 from approximately 11:54 AM PDT to 3:15 PM PDT, some Slack users experienced trouble receiving two-factor authentication (2FA) codes and therefore were unable to sign into Slack.
We determined this was caused by an upstream 2FA provider issue impacting code deliveries to US-based AT&T and Verizon devices. We monitored communications from the provider and confirmed Slack users could successfully sign in once the upstream issue was resolved.
<p>
<small>1:05pm PDT</small>
Our 2FA upstream provider is currently experiencing issues, which prevents Slack native 2FA code from arriving to US-based AT&T and Verizon devices.
We’re on the case and will provide another update once we have more details to share. We’re sorry for the disruption.
</p>
<p>
<small>1:36pm PDT</small>
We're actively investigating this issue and will be back with another update as soon as we have more information to share.
</p>
<p>
<small>2:20pm PDT</small>
Our upstream provider is reporting some signs of recovery. We will provide another update once we have more details to share.
We appreciate your patience while we sorted this out.
</p>
<p>
<small>3:42pm PDT</small>
Our upstream provider is reporting that the issue is now resolved. Users should now be able to receive 2FA codes on US-based AT&T and Verizon devices as expected.
Thank you again for your patience, and we apologize for the disruption.
</p>
<p>
<small>3:55pm PDT</small>
Issue summary:
On September 28, 2023 from approximately 11:54 AM PDT to 3:15 PM PDT, some Slack users experienced trouble receiving two-factor authentication (2FA) codes and therefore were unable to sign into Slack.
We determined this was caused by an upstream 2FA provider issue impacting code deliveries to US-based AT&T and Verizon devices. We monitored communications from the provider and confirmed Slack users could successfully sign in once the upstream issue was resolved.
</p>
https://slack-status.com/2023-09/94046787b0ca9008
2023-09-28T10:02:41-07:002023-09-28T17:49:59-07:00Incident: A small number of users experience latency with Slack
Issue summary:
From 8:00 AM PDT to 11:00 AM PDT on Thursday, September 28, 2023, a small number of users experienced delays in Slack, including sending and editing messages, and loading channels.
Upon investigation, we determined that a specific task in our backend was producing too many file deletion events and overloading our servers, resulting in these delays.
We implemented a fix to reduce the load on our servers and this resolved the issue for all affected users.
<p>
<small>10:02am PDT</small>
A small number of users are reporting latency with Slack, including delays in sending or editing messages.
We're investigating the issue and will share an update once we have more information.
</p>
<p>
<small>10:23am PDT</small>
We have determined that less than 1% of API calls are failing.
As this may impact certain actions like sending or editing messages, we recommend that users reattempt the action if it has failed. We hope to resolve this shortly and will be back with an update once new information becomes available.
</p>
<p>
<small>11:27am PDT</small>
Some users may be seeing signs of improvement, but we’re still monitoring the situation. We’ll be back once we have another update to share. Thank you for your patience as we sort this out.
</p>
<p>
<small>11:28am PDT</small>
We resolved the latency issues with Slack. Impacted actions like sending or editing messages have been restored. We appreciate you bearing with us as we worked to get things back to normal.
</p>
<p>
<small>5:49pm PDT</small>
Issue summary:
From 8:00 AM PDT to 11:00 AM PDT on Thursday, September 28, 2023, a small number of users experienced delays in Slack, including sending and editing messages, and loading channels.
Upon investigation, we determined that a specific task in our backend was producing too many file deletion events and overloading our servers, resulting in these delays.
We implemented a fix to reduce the load on our servers and this resolved the issue for all affected users.
</p>
https://slack-status.com/2023-09/054c6ac24cb629dd
2023-09-27T12:02:20-07:002023-09-28T14:54:37-07:00Incident: Create app on api.slack.com/apps not working
Issue summary:
On September 27, 2023 from 9:01 AM PDT to 12:32 PM PDT, some users experienced issues when attempting to create or edit custom apps on https://api.slack.com/apps, including seeing a "404 error" on this website.
We traced this back to a recent change that removed a function necessary for rendering a documentation page associated with the website.
We reverted the change, restoring the documentation page rendering component on https://api.slack.com/apps. Users should no longer have issues creating or editing custom apps and accessing this website.
Note: Due to an internal tooling error, we initially reported the times in this summary in UTC, but labeled with PDT. We have corrected this mistake and apologize for any confusion caused.
<p>
<small>12:02pm PDT</small>
We’re investigating a problem with creating apps on https://api.slack.com/apps. We’ll be back with another update as soon as we have more information to share.
</p>
<p>
<small>6:06pm PDT</small>
We’ve released a fix and users can create an app or workflow in their workspace once again.
We appreciate your patience while we sorted this out.
</p>
<p>
<small>2:54pm PDT</small>
Issue summary:
On September 27, 2023 from 9:01 AM PDT to 12:32 PM PDT, some users experienced issues when attempting to create or edit custom apps on https://api.slack.com/apps, including seeing a "404 error" on this website.
We traced this back to a recent change that removed a function necessary for rendering a documentation page associated with the website.
We reverted the change, restoring the documentation page rendering component on https://api.slack.com/apps. Users should no longer have issues creating or editing custom apps and accessing this website.
Note: Due to an internal tooling error, we initially reported the times in this summary in UTC, but labeled with PDT. We have corrected this mistake and apologize for any confusion caused.
</p>
https://slack-status.com/2023-09/badc5543a21e1fa7
2023-09-26T11:34:10-07:002023-09-27T10:24:03-07:00Incident: Can't dismiss Now you've got Later
Issue summary:
On September 26, 2023 from 10:00 AM PDT to 12:00 PM PDT, some users on our new interface design couldn't dismiss an information prompt that read "You now have Later".
These users also experienced their UI locking, and were unable to click anything in the Slack app. For some, the sparkle animation fixed over the Later icon in the sidebar also persisted longer than intended.
We identified the root cause as a missing detail in a recent code change. We reverted the change to mitigate the immediate issue, then deployed a fix to fully resolve the problem for all impacted users.
Note: Due to an internal tooling error, we initially reported the times in this summary in UTC, but labeled with PDT. We have corrected this mistake and apologize for any confusion caused.
<p>
<small>11:34am PDT</small>
We’re receiving reports around some users not being able to dismiss the "Now you've got Later" coach mark, and the inability to interact with the Slack app with their cursor.
We’ve identified the cause of this issue and we’re working on getting a fix out as soon as possible. We’ll be back once we have another update to share.
</p>
<p>
<small>11:55am PDT</small>
It's been reported that the sparkle animation over the Later icon in the sidebar is also lingering. Thank you for your continued patience.
</p>
<p>
<small>12:30pm PDT</small>
We’ve shipped a potential fix for this issue, but we’re still monitoring closely. We’ll let you know when functionality is fully restored.
</p>
<p>
<small>12:57pm PDT</small>
We’ve rolled out a fix and this issue should be resolved for all affected users.
If you’re still encountering any trouble, please reload Slack using Command + Shift + R (Mac) or Ctrl + Shift + R (Windows/Linux). Thank you for your patience while we sorted out this issue.
</p>
<p>
<small>2:29am PDT</small>
Issue summary:
On September 26, 2023 from 5:00pm PDT to around 7:00pm PDT, some customers may have experienced issues with an information prompt for the updated Later feature. In some cases, the prompt could not be dismissed and prevented access to other parts of Slack.
We identified the root cause as a missing detail in a recent code change. We reverted the change to mitigate the immediate issue, then deployed a fix to fully resolve the problem for all customers.
</p>
<p>
<small>10:24am PDT</small>
Issue summary:
On September 26, 2023 from 10:00 AM PDT to 12:00 PM PDT, some users on our new interface design couldn't dismiss an information prompt that read "You now have Later".
These users also experienced their UI locking, and were unable to click anything in the Slack app. For some, the sparkle animation fixed over the Later icon in the sidebar also persisted longer than intended.
We identified the root cause as a missing detail in a recent code change. We reverted the change to mitigate the immediate issue, then deployed a fix to fully resolve the problem for all impacted users.
Note: Due to an internal tooling error, we initially reported the times in this summary in UTC, but labeled with PDT. We have corrected this mistake and apologize for any confusion caused.
</p>
https://slack-status.com/2023-09/8315491ac04ad325
2023-09-24T17:17:04-07:002023-09-26T13:28:49-07:00Incident: Trouble with custom emoji
Issue summary:
From 12:54 PM PDT on September 22, 2023 to 09:13 AM PDT on September 25, 2023, custom emoji rendered as text for a small number of Enterprise Grid users.
We made a code change that inadvertently introduced unnecessary logging for routine emoji events. This unexpected logging increased traffic to the custom emoji infrastructure, impacting its performance.
We deployed a fix to remove the unnecessary logging, which reduced traffic to the custom emoji systems and restored normal emoji rendering.
A small fraction of the affected customers may have experienced lingering issues as the fix was gradually rolled out to everyone.
Note: Due to an internal tooling error, we initially reported the times in this summary in UTC, but labeled with PDT. We have corrected this mistake and apologize for any confusion caused.
<p>
<small>5:17pm PDT</small>
A small number of customers may be experiencing issues with custom emoji not rendering in messages or workflows. We are investigating and will be back with an update in 30 minutes.
We're very sorry for the disruption.
</p>
<p>
<small>5:48pm PDT</small>
There's no additional news to share just yet. We'll be back with another update in 30 minutes.
</p>
<p>
<small>6:19pm PDT</small>
We've established that this is only impacting a small number of Enterprise Grid organizations at this time.
We're still actively investigating, and will provide an update in 30 minutes.
</p>
<p>
<small>6:50pm PDT</small>
Our investigation is still ongoing. Thank you for bearing with us as we look in to this. We'll share another update in 30 minutes.
</p>
<p>
<small>7:19pm PDT</small>
We're still digging in to the cause of the custom emoji issue. We'll be back with an update in 30 minutes.
</p>
<p>
<small>7:48pm PDT</small>
We're still working on this, but we don't have any new information to share at this time. We'll provide another update as soon as we have more news.
</p>
<p>
<small>9:41am PDT</small>
We’ve identified the cause of this issue and we’re working on getting a fix out as soon as possible. We’ll be back once we have another update to share.
</p>
<p>
<small>11:29am PDT</small>
We’ve shipped a potential fix for this emoji issue, but we’re still monitoring closely. We’ll let you know when functionality is fully restored. Thank you for your patience.
</p>
<p>
<small>5:36pm PDT</small>
We've implemented a fix. While this has fully resolved the issue for many users, it's possible that a small number of customers may experience some continued impact until 10:30 AM PDT tomorrow, September 26, 2023.
We'll share an update here at that time. In the meantime, please write in to feedback@slack.com if you're still encountering issues, as our support team may be able to assist.
</p>
<p>
<small>10:26am PDT</small>
This issue is now resolved for all users. Thank you again for bearing with us in the meantime.
</p>
<p>
<small>1:28pm PDT</small>
Issue summary:
From 12:54 PM PDT on September 22, 2023 to 09:13 AM PDT on September 25, 2023, custom emoji rendered as text for a small number of Enterprise Grid users.
We made a code change that inadvertently introduced unnecessary logging for routine emoji events. This unexpected logging increased traffic to the custom emoji infrastructure, impacting its performance.
We deployed a fix to remove the unnecessary logging, which reduced traffic to the custom emoji systems and restored normal emoji rendering.
A small fraction of the affected customers may have experienced lingering issues as the fix was gradually rolled out to everyone.
Note: Due to an internal tooling error, we initially reported the times in this summary in UTC, but labeled with PDT. We have corrected this mistake and apologize for any confusion caused.
</p>
https://slack-status.com/2023-09/798ffab066d9a5d6
2023-09-21T09:08:44-07:002023-09-22T10:48:18-07:00Incident: Slack Connect invitations page not loading
Issue summary:
On September 21, 2023 from 8:00 PM PDT to 1:30 PM PDT, some users encountered a message stating "Something went wrong" when accessing the Slack Connect invites page.
We determined that rate limits set for querying invites were the cause of the "Something went wrong" message. These limits were set prior to the product decision to include these in the Activity view. Because use of this view has increased, the limits are reached much sooner than originally expected.
These limits were lifted for the invites page, resolving the issue for users. We appreciate your patience while we sorted this out and apologize for any inconvenience this may have caused.
Note: Due to an internal tooling error, we initially reported the times in this summary in UTC, but labeled with PDT. We have corrected this mistake and apologize for any confusion caused.
<p>
<small>9:08am PDT</small>
We’re receiving reports around trouble loading the Slack Connect invitations page. We’re on the case and will provide another update once we have more details to share. We’re sorry for the disruption.
</p>
<p>
<small>1:39pm PDT</small>
We’ve released a fix and all impacted features are up and running once again. We appreciate your patience while we sorted this out.
</p>
<p>
<small>10:48am PDT</small>
Issue summary:
On September 21, 2023 from 8:00 PM PDT to 1:30 PM PDT, some users encountered a message stating "Something went wrong" when accessing the Slack Connect invites page.
We determined that rate limits set for querying invites were the cause of the "Something went wrong" message. These limits were set prior to the product decision to include these in the Activity view. Because use of this view has increased, the limits are reached much sooner than originally expected.
These limits were lifted for the invites page, resolving the issue for users. We appreciate your patience while we sorted this out and apologize for any inconvenience this may have caused.
Note: Due to an internal tooling error, we initially reported the times in this summary in UTC, but labeled with PDT. We have corrected this mistake and apologize for any confusion caused.
</p>