Resolved
All remediating actions have been completed. Our support engineers are working directly with customers who were impacted during this incident.
Our team will follow up with a root cause analysis.
Posted Jan 12, 2023 - 14:11 UTC
Monitoring
Most deployments have been restored and we are monitoring closely. All impacted customers have been contacted and we're working closely with those impacted to ensure systems are functioning normally.
Posted Jan 12, 2023 - 06:16 UTC
Update
Restoration is still proceeding and support is reaching out directly to impacted customers with updates. We will continue to update with progress.
Posted Jan 12, 2023 - 03:23 UTC
Update
We have successfully restored a number of deployments. Restoration efforts are continuing, and we will continue to update as new information comes to light. Thank you for your patience.
Posted Jan 12, 2023 - 02:04 UTC
Update
We have begun restoring deployments from snapshot. We will update again in one hour.
Posted Jan 12, 2023 - 00:34 UTC
Update
Work continues on implementing the fix across regions. Another update will be sent in 1h.
Posted Jan 11, 2023 - 22:50 UTC
Update
Work continues on implementing the fix across regions. Another update will be sent in 1h.
Posted Jan 11, 2023 - 21:44 UTC
Update
Work continues on implementing the fix across regions. Another update will be sent in 1h.
Posted Jan 11, 2023 - 19:47 UTC
Update
Work continues on implementing the fix across regions. Another update will be sent in 1h.
Posted Jan 11, 2023 - 17:58 UTC
Update
Work continues on implementing the fix across regions. Another update will be sent in 1h.
Posted Jan 11, 2023 - 16:23 UTC
Update
Work continues on implementing the fix across regions. Another update will be sent in 1h.
Posted Jan 11, 2023 - 14:08 UTC
Update
We are progressing with the implementation of our fix and our engineers are focused on restoring any affected clusters.
Posted Jan 11, 2023 - 12:48 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jan 11, 2023 - 12:03 UTC
Identified
We have already reached out to all affected customers and are working on restoring the affected deployments.
Next update will be in 1hr.
Posted Jan 11, 2023 - 11:46 UTC
Update
We are continuing to investigate this issue.
Posted Jan 11, 2023 - 08:55 UTC
Update
We have confirmed which deployments got affected in which regions. We have blob storage snapshots and we will be attempting recovery from our backups.
Posted Jan 11, 2023 - 08:28 UTC
Update
We are continuing to investigate this issue.
Posted Jan 11, 2023 - 08:26 UTC
Update
We are continuing to investigate this issue.
Posted Jan 11, 2023 - 08:15 UTC
Update
We are continuing to investigate this issue.
Posted Jan 11, 2023 - 08:15 UTC
Investigating
We have detected an automation failure that caused some very limited number of customer deployments to be incorrectly deleted between 2UTC and 3UTC on the 01/11/2023. The impacted deployments are spread over multiple regions.
Posted Jan 11, 2023 - 08:05 UTC
This incident affected: AWS N. Virginia (us-east-1) (AWS EC2 Health: us-east-1, Snapshot Storage Infrastructure (S3): us-east-1, Deployment snapshots: AWS us-east-1, Deployment hosts: AWS us-east-1, Deployment logging: AWS us-east-1, Deployment metrics: AWS us-east-1, Deployment orchestration (Create/Edit/Restart/Delete): AWS us-east-1, Elasticsearch connectivity: AWS us-east-1, Kibana connectivity: AWS us-east-1, APM connectivity: AWS us-east-1, App Search connectivity: AWS us-east-1, Enterprise Search connectivity: AWS us-east-1, Watcher email actions), AWS N. California (us-west-1) (AWS EC2 Health: us-west-1, Snapshot Storage Infrastructure (S3): us-west-1, Deployment snapshots: AWS us-west-1, Deployment logging: AWS us-west-1, Deployment orchestration (Create/Edit/Restart/Delete): AWS us-west-1, Deployment metrics: AWS us-west-1, Deployment hosts: AWS us-west-1, Elasticsearch connectivity: AWS us-west-1, Kibana connectivity: AWS us-west-1, APM connectivity: AWS us-west-1, App Search connectivity: AWS us-west-1, Enterprise Search connectivity: AWS us-west-1, Watcher email actions), AWS Oregon (us-west-2) (AWS EC2 Health: us-west-2, Deployment logging: AWS us-west-2, Deployment metrics: AWS us-west-2, Deployment orchestration (Create/Edit/Restart/Delete): AWS us-west-2, Deployment snapshots: AWS us-west-2, Deployment hosts: AWS us-west-2, Elasticsearch connectivity: AWS us-west-2, Kibana connectivity: AWS us-west-2, APM connectivity: AWS us-west-2, App Search connectivity: AWS us-west-2, Enterprise Search connectivity: AWS us-west-2, Watcher email actions), AWS London (eu-west-2) (AWS EC2 Health: eu-west-2, Snapshot Storage Infrastructure (S3): eu-west-2, Deployment logging: AWS eu-west-2, Deployment orchestration (Create/Edit/Restart/Delete): AWS eu-west-2, Deployment hosts: AWS eu-west-2, Deployment snapshots: AWS eu-west-2, Elasticsearch connectivity: AWS eu-west-2, Kibana connectivity: AWS eu-west-2, APM connectivity: AWS eu-west-2, Deployment metrics: AWS eu-west-2, App Search connectivity: AWS eu-west-2, Enterprise Search connectivity: AWS eu-west-2, Watcher email actions), GCP Oregon (us-west1) (Deployment logging: GCP us-west1, Deployment snapshots: GCP us-west1, Deployment metrics: GCP us-west1, Deployment orchestration (Create/Edit/Restart/Delete): GCP us-west1, Deployment hosts: GCP us-west1, Elasticsearch connectivity: GCP us-west1, Kibana connectivity: GCP us-west1, APM connectivity: GCP us-west1, App Search connectivity: GCP us-west1, Enterprise Search connectivity: GCP us-west1, Watcher email actions), GCP Iowa (us-central1) (Deployment metrics: GCP us-central1, Deployment logging: GCP us-central1, Deployment snapshots: GCP us-central1, Deployment orchestration (Create/Edit/Restart/Delete): GCP us-central1, Deployment hosts: GCP us-central1, Elasticsearch connectivity: GCP us-central1, Kibana connectivity: GCP us-central1, APM connectivity: GCP us-central1, App Search connectivity: GCP us-central1, Enterprise Search connectivity: GCP us-central1, Watcher email actions), GCP Frankfurt (europe-west3) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west3, Deployment metrics: GCP europe-west3, Deployment hosts: GCP europe-west3, Deployment snapshots: GCP europe-west3, Deployment logging: GCP europe-west3, Elasticsearch connectivity: GCP europe-west3, Kibana connectivity: GCP europe-west3, APM connectivity: GCP europe-west3, App Search connectivity: GCP europe-west3, Enterprise Search connectivity: GCP europe-west3, Watcher email actions), GCP Belgium (europe-west1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west1, Deployment logging: GCP europe-west1, Deployment snapshots: GCP europe-west1, Deployment hosts: GCP europe-west1, Deployment metrics: GCP europe-west1, Kibana connectivity: GCP europe-west1, APM connectivity: GCP europe-west1, Elasticsearch connectivity: GCP europe-west1, App Search connectivity: GCP europe-west1, Enterprise Search connectivity: GCP europe-west1, Watcher email actions), GCP London (europe-west2) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west2, Deployment metrics: GCP europe-west2, APM connectivity: GCP europe-west2, Deployment snapshots: GCP europe-west2, Kibana connectivity: GCP europe-west2, Deployment logging: GCP europe-west2, Deployment hosts: GCP europe-west2, Elasticsearch connectivity: GCP europe-west2, App Search connectivity: GCP europe-west2, Enterprise Search connectivity: GCP europe-west2, Watcher email actions), Azure Virginia (azure-eastus2) (Deployment logging: Azure azure-eastus2, Elasticsearch connectivity: Azure azure-eastus2, Deployment metrics: Azure azure-eastus2, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-eastus2, APM connectivity: Azure azure-eastus2, Deployment hosts: Azure azure-eastus2, Deployment snapshots: Azure azure-eastus2, Kibana connectivity: Azure azure-eastus2, App Search connectivity: Azure azure-eastus2, Enterprise Search connectivity: Azure azure-eastus2, Watcher email actions), Azure Netherlands (azure-westeurope) (Deployment snapshots: Azure azure-westeurope, Deployment hosts: Azure azure-westeurope, Deployment metrics: Azure azure-westeurope, Deployment logging: Azure azure-westeurope, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-westeurope, APM connectivity: Azure azure-westeurope, Elasticsearch connectivity: Azure azure-westeurope, Kibana connectivity: Azure azure-westeurope, App Search connectivity: Azure azure-westeurope, Enterprise Search connectivity: Azure azure-westeurope, Watcher email actions), Azure Washington (azure-westus2) (Elasticsearch connectivity: Azure azure-westus2, Kibana connectivity: Azure azure-westus2, Deployment logging: Azure azure-westus2, Deployment snapshots: Azure azure-westus2, APM connectivity: Azure azure-westus2, Deployment hosts: Azure azure-westus2, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-westus2, Deployment metrics: Azure azure-westus2, App Search connectivity: Azure azure-westus2, Enterprise Search connectivity: Azure azure-westus2, Watcher email actions), GCP South Carolina (us-east1) (APM connectivity: GCP us-east1, Deployment hosts: GCP us-east1, Elasticsearch connectivity: GCP us-east1, Deployment metrics: GCP us-east1, Deployment logging: GCP us-east1, Kibana connectivity: GCP us-east1, Deployment snapshots: GCP us-east1, Deployment orchestration (Create/Edit/Restart/Delete): GCP us-east1, Enterprise Search connectivity: GCP us-east1, App Search connectivity: GCP us-east1, Watcher email actions), Azure London (azure-uksouth) (Deployment logging: Azure azure-uksouth, Deployment hosts: Azure azure-uksouth, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-uksouth, APM connectivity: Azure azure-uksouth, Deployment metrics: Azure azure-uksouth, Kibana connectivity: Azure azure-uksouth, Deployment snapshots: Azure azure-uksouth, Elasticsearch connectivity: Azure azure-uksouth, Enterprise Search connectivity: Azure azure-uksouth, App Search connectivity: Azure azure-uksouth, Watcher email actions), GCP Taiwan (asia-east1) (Deployment logging: GCP asia-east1, Deployment metrics: GCP asia-east1, APM connectivity: GCP asia-east1, Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-east1, Kibana connectivity: GCP asia-east1, Elasticsearch connectivity: GCP asia-east1, Deployment snapshots: GCP asia-east1, Deployment hosts: GCP asia-east1, App Search connectivity: GCP asia-east1, Enterprise Search connectivity: GCP asia-east1, Watcher email actions), Azure Iowa (azure-centralus) (Deployment snapshots: Azure azure-centralus, Deployment metrics: Azure azure-centralus, Kibana connectivity: Azure azure-centralus, Deployment logging: Azure azure-centralus, Elasticsearch connectivity: Azure azure-centralus, APM connectivity: Azure azure-centralus, Deployment hosts: Azure azure-centralus, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-centralus, App Search connectivity: Azure azure-centralus, Enterprise Search connectivity: Azure azure-centralus, Watcher email actions), Azure Ireland (azure-northeurope) (APM connectivity: Azure azure-northeurope, Deployment metrics: Azure azure-northeurope, App Search connectivity: Azure azure-northeurope, Deployment snapshots: Azure azure-northeurope, Enterprise Search connectivity: Azure azure-northeurope, Deployment logging: Azure azure-northeurope, Watcher email actions, Kibana connectivity: Azure azure-northeurope, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-northeurope, Elasticsearch connectivity: Azure azure-northeurope, Deployment hosts: Azure azure-northeurope), Azure France (azure-francecentral) (Deployment logging: Azure azure-francecentral, Deployment hosts: Azure azure-francecentral, Watcher email actions, App Search connectivity: Azure azure-francecentral, Deployment metrics: Azure azure-francecentral, Enterprise Search connectivity: Azure azure-francecentral, Kibana connectivity: Azure azure-francecentral, Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-francecentral, Deployment snapshots: Azure azure-francecentral, Elasticsearch connectivity: Azure azure-francecentral, APM connectivity: Azure azure-francecentral), and GCP Seoul (asia-northeast3) (Deployment hosts: GCP asia-northeast3, Kibana connectivity: GCP asia-northeast3, App Search connectivity: GCP asia-northeast3, APM connectivity: GCP asia-northeast3, Deployment metrics: GCP asia-northeast3, Deployment snapshots: GCP asia-northeast3, Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-northeast3, Deployment logging: GCP asia-northeast3, Enterprise Search connectivity: GCP asia-northeast3, Watcher email actions, Elasticsearch connectivity: GCP asia-northeast3).