Resolved
All remediating actions have been completed. Our support engineers are working directly with customers who were impacted during this incident.
Our team will follow up with a root cause analysis.
Posted Jan 12, 2023 - 14:11 UTC
Monitoring
Most deployments have been restored and we are monitoring closely. All impacted customers have been contacted and we're working closely with those impacted to ensure systems are functioning normally.
Posted Jan 12, 2023 - 06:16 UTC
Update
Restoration is still proceeding and support is reaching out directly to impacted customers with updates. We will continue to update with progress.
Posted Jan 12, 2023 - 03:23 UTC
Update
We have successfully restored a number of deployments. Restoration efforts are continuing, and we will continue to update as new information comes to light. Thank you for your patience.
Posted Jan 12, 2023 - 02:04 UTC
Update
We have begun restoring deployments from snapshot. We will update again in one hour.
Posted Jan 12, 2023 - 00:34 UTC
Update
Work continues on implementing the fix across regions. Another update will be sent in 1h.
Posted Jan 11, 2023 - 22:50 UTC
Update
Work continues on implementing the fix across regions. Another update will be sent in 1h.
Posted Jan 11, 2023 - 21:44 UTC
Update
Work continues on implementing the fix across regions. Another update will be sent in 1h.
Posted Jan 11, 2023 - 19:47 UTC
Update
Work continues on implementing the fix across regions. Another update will be sent in 1h.
Posted Jan 11, 2023 - 17:58 UTC
Update
Work continues on implementing the fix across regions. Another update will be sent in 1h.
Posted Jan 11, 2023 - 16:23 UTC
Update
Work continues on implementing the fix across regions. Another update will be sent in 1h.
Posted Jan 11, 2023 - 14:08 UTC
Update
We are progressing with the implementation of our fix and our engineers are focused on restoring any affected clusters.
Posted Jan 11, 2023 - 12:48 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jan 11, 2023 - 12:03 UTC
Identified
We have already reached out to all affected customers and are working on restoring the affected deployments.
Next update will be in 1hr.
Posted Jan 11, 2023 - 11:46 UTC
Update
We are continuing to investigate this issue.
Posted Jan 11, 2023 - 08:55 UTC
Update
We have confirmed which deployments got affected in which regions. We have blob storage snapshots and we will be attempting recovery from our backups.
Posted Jan 11, 2023 - 08:28 UTC
Update
We are continuing to investigate this issue.
Posted Jan 11, 2023 - 08:26 UTC
Update
We are continuing to investigate this issue.
Posted Jan 11, 2023 - 08:15 UTC
Update
We are continuing to investigate this issue.
Posted Jan 11, 2023 - 08:15 UTC
Investigating
We have detected an automation failure that caused some very limited number of customer deployments to be incorrectly deleted between 2UTC and 3UTC on the 01/11/2023. The impacted deployments are spread over multiple regions.
Posted Jan 11, 2023 - 08:05 UTC
This incident affected: AWS N. Virginia (us-east-1) (AWS EC2 Health: us-east-1, Snapshot Storage Infrastructure (S3): us-east-1, AWS Infrastructure health: us-east-1, Deployment orchestration (Create/Edit/Restart/Delete): AWS us-east-1, Elasticsearch connectivity: AWS us-east-1, Kibana connectivity: AWS us-east-1, APM connectivity: AWS us-east-1), AWS N. California (us-west-1) (AWS EC2 Health: us-west-1, Snapshot Storage Infrastructure (S3): us-west-1, AWS Infrastructure health: us-west-1, Deployment orchestration (Create/Edit/Restart/Delete): AWS us-west-1, Elasticsearch connectivity: AWS us-west-1, Kibana connectivity: AWS us-west-1, APM connectivity: AWS us-west-1), AWS Oregon (us-west-2) (AWS EC2 Health: us-west-2, Deployment orchestration (Create/Edit/Restart/Delete): AWS us-west-2, AWS Infrastructure health: us-west-2, Elasticsearch connectivity: AWS us-west-2, Kibana connectivity: AWS us-west-2, APM connectivity: AWS us-west-2), AWS London (eu-west-2) (AWS EC2 Health: eu-west-2, Snapshot Storage Infrastructure (S3): eu-west-2, Deployment orchestration (Create/Edit/Restart/Delete): AWS eu-west-2, AWS Infrastructure health: eu-west-2, Elasticsearch connectivity: AWS eu-west-2, Kibana connectivity: AWS eu-west-2, APM connectivity: AWS eu-west-2), GCP Frankfurt (europe-west3) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west3, GCP Infrastructure health: europe-west3, Elasticsearch connectivity: GCP europe-west3, Kibana connectivity: GCP europe-west3, APM connectivity: GCP europe-west3), GCP Belgium (europe-west1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west1, GCP Infrastructure health: europe-west1, Kibana connectivity: GCP europe-west1, APM connectivity: GCP europe-west1, Elasticsearch connectivity: GCP europe-west1), GCP London (europe-west2) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west2, APM connectivity: GCP europe-west2, GCP Infrastructure health: europe-west2, Kibana connectivity: GCP europe-west2, Elasticsearch connectivity: GCP europe-west2), Azure Netherlands (azure-westeurope) (Azure Infrastructure health: azure-westeurope, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-westeurope, APM connectivity: Azure azure-westeurope, Elasticsearch connectivity: Azure azure-westeurope, Kibana connectivity: Azure azure-westeurope), Azure Washington (azure-westus2) (Elasticsearch connectivity: Azure azure-westus2, Kibana connectivity: Azure azure-westus2, Azure Infrastructure health: azure-westus2, APM connectivity: Azure azure-westus2, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-westus2), GCP South Carolina (us-east1) (APM connectivity: GCP us-east1, Elasticsearch connectivity: GCP us-east1, Kibana connectivity: GCP us-east1, GCP Infrastructure health: us-east1, Deployment orchestration (Create/Edit/Restart/Delete): GCP us-east1), Azure Iowa (azure-centralus) (Azure Infrastructure health: azure-centralus, Kibana connectivity: Azure azure-centralus, Elasticsearch connectivity: Azure azure-centralus, APM connectivity: Azure azure-centralus, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-centralus), Azure Ireland (azure-northeurope) (APM connectivity: Azure azure-northeurope, Azure Infrastructure health: azure-northeurope, Kibana connectivity: Azure azure-northeurope, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-northeurope, Elasticsearch connectivity: Azure azure-northeurope), GCP Oregon (us-west1) (CSP Infrastructure Health: GCP us-west1, Deployment orchestration (Create/Edit/Restart/Delete): GCP us-west1, Elasticsearch connectivity: GCP us-west1, Kibana connectivity: GCP us-west1, APM connectivity: GCP us-west1), Azure Virginia (azure-eastus2) (Elasticsearch connectivity: Azure azure-eastus2, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-eastus2, APM connectivity: Azure azure-eastus2, Azure Infrastructure health: azure-eastus2, Kibana connectivity: Azure azure-eastus2), GCP Seoul (asia-northeast3) (Kibana connectivity: GCP asia-northeast3, APM connectivity: GCP asia-northeast3, GCP Infrastructure health: asia-northeast3, Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-northeast3, Elasticsearch connectivity: GCP asia-northeast3), Azure London (azure-uksouth) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-uksouth, APM connectivity: Azure azure-uksouth, Kibana connectivity: Azure azure-uksouth, Azure Infrastructure health: azure-uksouth, Elasticsearch connectivity: Azure azure-uksouth), GCP Taiwan (asia-east1) (APM connectivity: GCP asia-east1, Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-east1, Kibana connectivity: GCP asia-east1, Elasticsearch connectivity: GCP asia-east1, GCP Infrastructure health: asia-east1), GCP Iowa (us-central1) (GCP Infrastructure health: us-central1, Deployment orchestration (Create/Edit/Restart/Delete): GCP us-central1, Elasticsearch connectivity: GCP us-central1, Kibana connectivity: GCP us-central1, APM connectivity: GCP us-central1), and Azure France (azure-francecentral) (Kibana connectivity: Azure azure-francecentral, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-francecentral, Azure Infrastructure health: azure-francecentral, Elasticsearch connectivity: Azure azure-francecentral, APM connectivity: Azure azure-francecentral).