Customer logs and metrics appear to be flowing again, with queues trending downwards. We expect cluster logging and metric data to be up to date within the next hour.
One of our backend monitoring clusters had entered a degraded state which was impacting indexing rates within our Logstash consumers. We're still investigating what caused this cluster to get into this state, but believe two nodes were impacted by our recent hot-warm incident
resulting in those nodes not able to keep up with cluster state changes. We've restored this cluster to an operational state, which in turn restored our log ingestion rates.
We'll post another update in 30 minutes, or as new information comes to hand.