Scheduled snapshots not running on some deployments
Resolved
The fix has been rolled out to all regions and verified. Thanks for your patience!
Posted Apr 27, 2020 - 20:40 UTC
Identified
We have confirmed that our fix works and are working on rolling it out to all production environments. Once that's complete and we've verified the fix, we'll post another update. We expect this to take approximately two hours.
Posted Apr 27, 2020 - 18:07 UTC
Update
We're still working on testing the fix in our staging environment and canary regions. We will have another update in one hour.
Posted Apr 27, 2020 - 17:21 UTC
Investigating
Regularly scheduled snapshots are not happening on approximately 2% of cloud deployments across all regions and cloud providers. Larger deployments are impacted. We have a fix in progress and will be rolling it to production after testing.
We will post an update in one hour.
Posted Apr 27, 2020 - 16:04 UTC
This incident affected: Azure Netherlands (azure-westeurope) (Deployment snapshots: Azure azure-westeurope), GCP Mumbai (asia-south1) (Deployment snapshots: GCP asia-south1), GCP Singapore (asia-southeast1) (Deployment snapshots: GCP asia-southeast1), GCP Oregon (us-west1) (Deployment snapshots: GCP us-west1), AWS Sydney (ap-southeast-2) (Deployment snapshots: AWS ap-southeast-2), AWS N. Virginia (us-east-1) (Deployment snapshots: AWS us-east-1), Azure Singapore (azure-southeastasia) (Deployment snapshots: Azure azure-southeastasia), GCP Belgium (europe-west1) (Deployment snapshots: GCP europe-west1), AWS Singapore (ap-southeast-1) (Deployment snapshots: AWS ap-southeast-1), AWS N. California (us-west-1) (Deployment snapshots: AWS us-west-1), GCP London (europe-west2) (Deployment snapshots: GCP europe-west2), Azure Washington (azure-westus2) (Deployment snapshots: Azure azure-westus2), GCP São Paulo (southamerica-east1) (Deployment snapshots: GCP southamerica-east1), GCP Iowa (us-central1) (Deployment snapshots: GCP us-central1), GCP Frankfurt (europe-west3) (Deployment snapshots: GCP europe-west3), AWS Tokyo (ap-northeast-1) (Deployment snapshots: AWS ap-northeast-1), AWS São Paulo (sa-east-1) (Deployment snapshots: AWS sa-east-1), GCP Montreal (northamerica-northeast1) (Deployment snapshots: GCP northamerica-northeast1), Azure Tokyo (azure-japaneast) (Deployment snapshots: Azure azure-japaneast), GCP Tokyo (asia-northeast1) (Deployment snapshots: GCP asia-northeast1), Azure Virginia (azure-eastus2) (Deployment snapshots: Azure azure-eastus2), Azure London (azure-uksouth) (Deployment snapshots: Azure azure-uksouth), AWS Oregon (us-west-2) (Deployment snapshots: AWS us-west-2), GCP Sydney (australia-southeast1) (Deployment snapshots: GCP australia-southeast1), GCP South Carolina (us-east1) (Deployment snapshots: GCP us-east1), AWS Frankfurt (eu-central-1) (Deployment snapshots: AWS eu-central-1), AWS London (eu-west-2) (Deployment snapshots: AWS eu-west-2), and AWS Ireland (eu-west-1) (Deployment snapshots: AWS eu-west-1).