This incident is now resolved. All mail is enabled and flowing properly. Thank you for your patience as we investigated and resolved this issue.
Posted Apr 09, 2019 - 23:32 UTC
Monitoring
We have cleared the mail backlog, deleting out of memory and cluster problem emails up until 10:30 UTC. Any clusters with ongoing issues will re-alert, sending users a fresh notification. All mail is enabled and flowing properly. We'll continue to monitor mail queues for any further delays.
Posted Apr 09, 2019 - 22:52 UTC
Update
We have identified what we believe is the root cause and pushed a partial fix. Password reset, new trial, watcher, and most other routine emails are being sent and their backlog has cleared. Out of memory and cluster problem emails are still affected, we are preparing a fix and hope to have these flowing again soon.
As part of this fix, we'll be deleting cluster problem emails from 19:17 UTC. Clusters with ongoing issues will receive receive new notifications when mail is flowing again.
Posted Apr 09, 2019 - 22:29 UTC
Update
We have identified what we believe is the root cause and pushed a partial fix. Password reset, new trial, watcher, and most other routine emails are being sent and their backlog has cleared. Out of memory and cluster problem emails are still affected, and we are working on a fix.
Posted Apr 09, 2019 - 21:52 UTC
Identified
The fix we implemented did not resolve the issue, so at this time, mail is still delayed. We are investigating other options and hope to have this resolved soon.
Posted Apr 09, 2019 - 21:12 UTC
Monitoring
We’ve identified the issue and pushed a fix. The mail backlog has been processed and all mails should be going out as usual. We will continue to monitor.
Posted Apr 09, 2019 - 20:23 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Apr 09, 2019 - 20:01 UTC
Investigating
We're currently experiencing email deliverability issues for system-generated alerts, including but not limited to: disk space, CPU usage, OOM events and all other watcher generated alerts.
Additionally, this is resulting in a delay for trial signup and verification emails, but you should receive these once the deliverability issues are resolved.