Rebalance of our internal logging cluster is completed. All deployment logs have been restored.
The Deployment Performance graphs on cloud.elastic.co showing number of requests, search response times and indexing response may continue to show gaps during the span of this incident. These will be stable going forward.
Incident resolved.
Posted Mar 11, 2021 - 04:41 UTC
Monitoring
We were able to remediate issue related to fast growing of an ingest pipeline queues. Performance data for number of requests, search response times and indexing response times in the user console for period from March 10th 22:00 UTC till March 11th 01:00 UTC might be absent. Additionally we are expecting delays in logs and performance data delivery until cluster rebalance will be completed. We will monitor the situation and will post an update as a new information becomes available.
Posted Mar 11, 2021 - 01:50 UTC
Update
We're continuing to investigate. Deployment logging in AWS us-east-1 is the only component currently experiencing problems.
Posted Mar 10, 2021 - 23:07 UTC
Investigating
We're investigating an alert that our ingest pipeline queues are growing faster than being processed. We can see that events are still being processed. However, you may still see delayed logs and metrics for deployments in the Cloud Console.
Posted Mar 10, 2021 - 22:42 UTC
This incident affected: Global services (Cloud console) and AWS N. Virginia (us-east-1) (Deployment logging: AWS us-east-1).