We've rolled out the remediation to the affected regions and confirmed that everything is working as expected. Approximately 20 - 30% of the APM deployments in each region were impacted. We will be monitoring for the next hour and then will close out this incident.
Posted Feb 24, 2020 - 17:00 UTC
Update
We have identified this issue in other regions. GCP London (europe-west2), Azure Tokyo (azure-japaneast), GCP Mumbai (asia-south1), GCP Montreal (northamerica-northeast1), GCP N. Virginia (us-east4), GCP Sydney (australia-southeast1), AWS London (eu-west-2), AWS Tokyo (ap-northeast-1), Azure Virginia (azure-eastus2), Azure Singapore (azure-southeastasia), Azure Netherlands (azure-westeurope).
We have tested a proposed remediation strategy and have begun its deployment across the affected regions. APM service will be restored shortly. We are continuing to monitor the situation closely. Further updates will be posted within an hour.
Posted Feb 24, 2020 - 16:16 UTC
Identified
We have identified the cause of the connectivity issue. There is a discrepancy in our coordination layer that prevents routing of traffic to affected APM endpoints. We are assessing the best remediation approach and we will update within the next hour.
Posted Feb 24, 2020 - 15:07 UTC
Investigating
We are investigating APM connectivity issues in GCP Oregon (us-west1) and Frankfurt (europe-west3). Access to APM endpoint in those regions may be affected. Access to Elasticsearch and Kibana endpoints is operating normally. We will update within the next 30 minutes.
Posted Feb 24, 2020 - 14:11 UTC
This incident affected: GCP N. Virginia (us-east4) (APM connectivity: GCP us-east4), GCP London (europe-west2) (APM connectivity: GCP europe-west2), GCP Montreal (northamerica-northeast1) (APM connectivity: GCP northamerica-northeast1), GCP Sydney (australia-southeast1) (APM connectivity: GCP australia-southeast1), Azure Virginia (azure-eastus2) (APM connectivity: Azure azure-eastus2), Azure Netherlands (azure-westeurope) (APM connectivity: Azure azure-westeurope), GCP Mumbai (asia-south1) (APM connectivity: GCP asia-south1), Azure Singapore (azure-southeastasia) (APM connectivity: Azure azure-southeastasia), Azure Tokyo (azure-japaneast) (APM connectivity: Azure azure-japaneast), GCP Frankfurt (europe-west3) (APM connectivity: GCP europe-west3), AWS Tokyo (ap-northeast-1) (APM connectivity: AWS ap-northeast-1), GCP Oregon (us-west1) (APM connectivity: GCP us-west1), and AWS London (eu-west-2) (APM connectivity: AWS eu-west-2).