Engineers are working on follow-up items as the final part of our recovery procedure, and we are monitoring the status.
Posted Oct 28, 2020 - 10:02 UTC
Update
We have successfully recovered the ZooKeeper ensemble powering our coordination layer in GCP europe-west3. We are working towards getting back to normal state in that region.
Creation of new deployments is now working as expected. Topology and configuration updates to existing deployments in GCP europe-west3 are now working as expected.
Posted Oct 28, 2020 - 09:37 UTC
Identified
We are working towards recovering the ZooKeeper ensemble powering our coordination layer in GCP europe-west3. NVMe errors were observed in 2 out of 3 availability zones, and impacted the availability of the ZooKeeper quorum.
Creation of new deployments is still impaired. Topology and configuration updates to existing deployments in GCP europe-west3 is currently failing.
Next update within an hour.
Posted Oct 28, 2020 - 09:05 UTC
Update
We are continuing to investigate this issue. Creation of new deployments is still impaired. In addition, topology and configuration updates to existing deployments in GCP europe-west3 is currently failing. Next update within an hour.
Posted Oct 28, 2020 - 07:51 UTC
Investigating
We have discovered an issue in cloud that is preventing customers from creating new deployments in all regions. Related to this issue, existing deployments in gcp-europe-west3 are unable to be edited. Traffic to existing deployments globally are not affected. We will update as new information becomes available.
Posted Oct 28, 2020 - 05:57 UTC
This incident affected: Global services (Cloud console).