All affected deployments have been repaired. We are continuing to monitor.
Posted Jan 04, 2020 - 02:07 UTC
Identified
Remediation is approximately 50% completed, and is still in progress. We will update when remediation is completed.
Posted Jan 04, 2020 - 00:49 UTC
Update
Remediation is still in progress and we're monitoring for reoccurring for abnormal behavior. Note: Affected clusters will have their previous snopshots and snapshot histories reset. We'll plan on another update before 01:00 UTC.
Posted Jan 03, 2020 - 23:46 UTC
Update
Snapshots are working again for single-node clusters. We are working on remediation for the remaining ones. We'll provide a further a update by 01:00 UTC.
Posted Jan 03, 2020 - 23:12 UTC
Update
We have disabled suspect internal services that may be causing the snapshot failures. We're currently monitoring to see if the issues reoccur. In the meantime we're working on a remediation plan for the affected deployments. We'll provide an update at We'll update again before 00:30 UTC.
Posted Jan 03, 2020 - 22:27 UTC
Update
Investigation is still underway. The failures started recently, but we're still looking for a root cause. We'll update again before 22:30 UTC.
Posted Jan 03, 2020 - 20:42 UTC
Update
We're still investigating the source of the failures. We'll provide another update before 8:30 UTC.
Posted Jan 03, 2020 - 19:38 UTC
Investigating
We are currently investigating cluster snapshot failures over the last 24 hours in the Azure EastUS2 region. Clusters are otherwise still functional, but possibly missing the most recent snapshot(s). We'll provide an update in the next 30 minutes.
Posted Jan 03, 2020 - 19:05 UTC
This incident affected: Azure Virginia (azure-eastus2) (Deployment snapshots: Azure azure-eastus2).