High rate of failure when upgrading deployments to version 8.0

Incident Report for ESS (Public)

Resolved

There have been no further issues associated with this incident. Please reach out to support if you believe your deployment has been effected.
Posted Feb 25, 2022 - 03:53 UTC

Monitoring

AWS regions are fully deployed. At this point all regions across all providers have remediation in place. We are continuing to monitor impact from the issue.
Posted Feb 24, 2022 - 23:25 UTC

Update

Azure, GCP, and IBM regions are fully deployed, and remaining AWS regions are in progress. We are continuing to monitor impact from the issue. We'll update again within an hour.
Posted Feb 24, 2022 - 22:27 UTC

Update

Azure and GCP regions are fully deployed, and remaining AWS regions are in progress. We are continuing to monitor impact from the issue. We'll update again within an hour.
Posted Feb 24, 2022 - 21:31 UTC

Update

Production regions remains in progress, and we are continuing to monitor impact from the issue. We'll update again within an hour.
Posted Feb 24, 2022 - 20:35 UTC

Update

We're in the process of deploying a fix to all production regions and are continuing to monitor impact from the issue.
Posted Feb 24, 2022 - 19:38 UTC

Identified

We've identified some issues when applying settings to 8.0 instances and are deploying a fix targeted to resolve that impact while continuing the investigation to determine any other causes. We'll update again within two hours.
Posted Feb 24, 2022 - 15:40 UTC

Investigating

We're currently seeing a high rate of errors in deployments upgraded to version 8.0. We're currently investigating this issue and will update this status page within the two next hour, or as more information comes to hand.

There are no general cluster connectivity issues.
Posted Feb 24, 2022 - 13:38 UTC
This incident affected: Global services (Deployment orchestration (Create/Edit/Restart/Delete)).