Connectivity interruption for multiple GCP regions

Incident Report for ESS (Public)

Resolved

At this time all services are operating normally and we are marking this resolved. If you see any additional issues on your end please don't hesitate to reach out to our support team and we would like to thank you for your patience during this!

Posted Mar 27, 2020 - 22:26 UTC

Monitoring

We have applied mitigation on our end to stabilize performance and services as a result of the GCP networking issues. At this time we're monitoring for additional problems and will have another update within the hour. Please don't hesitate to reach out to us through our support team if you're still seeing issues and we thank you for your patience.

Posted Mar 27, 2020 - 21:52 UTC

Identified

We are continuing with mitigation efforts on our end and we're seeing improvements with control plane and cluster management functionality in the affected regions as Google implements their own mitigation efforts.

Google Status: https://status.cloud.google.com/incident/cloud-networking/20004

Posted Mar 27, 2020 - 21:07 UTC

Update

We are continuing the investigation on our end, however it appears this may be related to networking issues on the GCP side.

Google Status: https://status.cloud.google.com/incident/cloud-networking/20004

Posted Mar 27, 2020 - 20:41 UTC

Investigating

We are currently investigating an issue with connectivity to multiple GCP regions which is impacting CRUD operations for all clusters in the affected regions.

The GCP regions we have confirmed to be impacted are as follows and we'll update this list if it changes:

europe-west1, europe-west2, europe-west3, northamerica-northeast1, us-east1, us-east4

At this time this does not appear to be impacting services within AWS or Azure.

Posted Mar 27, 2020 - 20:13 UTC

This incident affected: Global services (Cloud console), GCP Frankfurt (europe-west3) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west3), GCP Belgium (europe-west1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west1), GCP London (europe-west2) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west2), GCP Montreal (northamerica-northeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP northamerica-northeast1), and GCP N. Virginia (us-east4) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-east4).