Lag in deployment logs and metrics in Cloud console
Incident Report for ESS (Public)
Resolved
Metrics and logs for deployments in all regions are no longer delayed and are accessible through the Cloud console.
Posted May 30, 2020 - 23:54 UTC
Monitoring
Delays in Azure Netherlands (azure-westeurope) has been cleared and will continue to be monitored.
Metrics and logs of deployments in that region are no longer lagging behind, and are accessible through the Cloud console.
Posted May 30, 2020 - 23:40 UTC
Identified
Delays in Azure Virginia (azure-eastus2) and GCP Tokyo (gcp-asia-northeast1) have been cleared and will continue to be monitored.
We have identified a fix and applying mitigations for to Azure Netherlands (azure-westeurope).
Posted May 30, 2020 - 22:59 UTC
Update
We are continuing to investigate delays in GCP Tokyo (gcp-asia-northeast1), Azure Netherlands (azure-westeurope), Azure Virginia (azure-eastus2).
Posted May 30, 2020 - 22:00 UTC
Update
We are continuing to investigate delays in GCP Tokyo (gcp-asia-northeast1), Azure Netherlands (azure-westeurope), Azure Virginia (azure-eastus2).
Delays in GCP Singapore (gcp-asia-southeast1) have been cleared and are continuing to be monitored.
Posted May 30, 2020 - 21:09 UTC
Update
We are continuing to investigate delays in GCP Singapore (gcp-asia-southeast1), Azure Netherlands (azure-westeurope), Azure Virginia (azure-eastus2)
Posted May 30, 2020 - 19:54 UTC
Update
Some of our mitigating actions have drained queues in GCP Mumbai (gcp-asia-south1).
Metrics and logs of deployments in that region are no longer lagging behind, and are accessible through the Cloud console.

Our engineers are working on further mitigations while investigating contributing factors for the delays in GCP Singapore (gcp-asia-southeast1), Azure Netherlands (azure-westeurope), Azure Virginia (azure-eastus2)
Posted May 30, 2020 - 18:57 UTC
Update
We are continuing to investigate this issue.
Posted May 30, 2020 - 17:54 UTC
Update
We are continuing the investigation. We will provide an update in the next 1 hour.
Posted May 30, 2020 - 16:23 UTC
Update
We are continuing the investigation. We will provide an update in the next 1 hour.
Posted May 30, 2020 - 15:26 UTC
Investigating
We are observing ingestion queues in our internal metrics and logging infrastructure.
Customers may experience delayed visibility into the clusters logs and graphs displayed in the UI.
Engineers are actively investigating the cause of the delay and we will provide an update in the next 1 hour.
Posted May 30, 2020 - 14:25 UTC
This incident affected: Azure Virginia (azure-eastus2) (Deployment logging: Azure azure-eastus2, Deployment metrics: Azure azure-eastus2), Azure Netherlands (azure-westeurope) (Deployment metrics: Azure azure-westeurope, Deployment logging: Azure azure-westeurope), and GCP Tokyo (asia-northeast1) (Deployment metrics: GCP asia-northeast1, Deployment logging: GCP asia-northeast1).