Deployment creation and update may fail
Incident Report for ESS (Public)
Resolved
This incident has been resolved.
Posted Apr 29, 2021 - 06:06 UTC
Monitoring
We have applied mitigations in GCP regions now as well. This concludes the mitigation work. We'll continue to monitor for the next 3 hours before resolving.
Posted Apr 29, 2021 - 03:00 UTC
Update
We have applied the mitigation on the all AWS regions.

GCP regions are still in progress.
Posted Apr 28, 2021 - 23:52 UTC
Update
We have applied the mitigation on the following AWS regions:
ap-east-1
ap-northeast-1
ap-northeast-2
ap-southeast-1
ap-southeast-2
eu-west-1
eu-west-2
us-east-1

The rest of AWS regions are pending.

We have started to apply mitigation with us-west2 region on GCP.
Posted Apr 28, 2021 - 21:59 UTC
Update
We have applied mitigations that prevent the issue from happening across Azure regions, and are working on applying these to AWS now.

Due to differences in configuration of our infrastructure on GCP, we are still working through the process of applying the same mitigation on GCP.
Posted Apr 28, 2021 - 20:03 UTC
Identified
We're encountering issues on deployment creation and update. Users may see their new deployment failing to create or their deployment changes failing to complete.

We're identified that this is due to a Docker bug and we are currently working on mitigating the issue.

There are no general cluster connectivity issues.

We will update this status page within the next four hours, or as more information comes to hand.
Posted Apr 28, 2021 - 11:51 UTC
This incident affected: Global services (Deployment orchestration (Create/Edit/Restart/Delete)).