Resolved
The fix has been rolled out to all regions and verified. Thanks for your patience!
Posted Apr 27, 2020 - 20:40 UTC
Identified
We have confirmed that our fix works and are working on rolling it out to all production environments. Once that's complete and we've verified the fix, we'll post another update. We expect this to take approximately two hours.
Posted Apr 27, 2020 - 18:07 UTC
Update
We're still working on testing the fix in our staging environment and canary regions. We will have another update in one hour.
Posted Apr 27, 2020 - 17:21 UTC
Investigating
Regularly scheduled snapshots are not happening on approximately 2% of cloud deployments across all regions and cloud providers. Larger deployments are impacted. We have a fix in progress and will be rolling it to production after testing.
We will post an update in one hour.
Posted Apr 27, 2020 - 16:04 UTC
This incident affected: Azure Netherlands (azure-westeurope) (Azure Infrastructure health: azure-westeurope), GCP Mumbai (asia-south1) (GCP Infrastructure health: asia-south1), GCP Singapore (asia-southeast1) (GCP Infrastructure health: asia-southeast1), GCP Oregon (us-west1) (CSP Infrastructure Health: GCP us-west1), AWS Sydney (ap-southeast-2) (AWS Infrastructure health: ap-southeast-2), AWS N. Virginia (us-east-1) (AWS Infrastructure health: us-east-1), Azure Singapore (azure-southeastasia) (Deployment snapshots: Azure azure-southeastasia), GCP Belgium (europe-west1) (GCP Infrastructure health: europe-west1), AWS Singapore (ap-southeast-1) (Deployment snapshots: AWS ap-southeast-1), AWS N. California (us-west-1) (AWS Infrastructure health: us-west-1), GCP London (europe-west2) (GCP Infrastructure health: europe-west2), Azure Washington (azure-westus2) (Azure Infrastructure health: azure-westus2), GCP São Paulo (southamerica-east1) (GCP Infrastructure health: southamerica-east1), GCP Iowa (us-central1) (GCP Infrastructure health: us-central1), GCP Frankfurt (europe-west3) (GCP Infrastructure health: europe-west3), AWS Tokyo (ap-northeast-1) (AWS Infrastructure health: ap-northeast-1), AWS São Paulo (sa-east-1) (AWS Infrastructure health: sa-east-1), GCP Montreal (northamerica-northeast1) (GCP Infrastructure health: northamerica-northeast1), Azure Tokyo (azure-japaneast) (Azure Infrastructure health: azure-japaneast), GCP Tokyo (asia-northeast1) (GCP Infrastructure health: asia-northeast1), Azure Virginia (azure-eastus2) (Azure Infrastructure health: azure-eastus2), Azure London (azure-uksouth) (Azure Infrastructure health: azure-uksouth), AWS Oregon (us-west-2) (AWS Infrastructure health: us-west-2), GCP Sydney (australia-southeast1) (GCP Infrastructure health: australia-southeast1), GCP South Carolina (us-east1) (GCP Infrastructure health: us-east1), AWS Frankfurt (eu-central-1) (AWS Infrastructure health: eu-central-1), AWS London (eu-west-2) (AWS Infrastructure health: eu-west-2), and AWS Ireland (eu-west-1) (AWS Infrastructure health: eu-west-1).