Pre-emptive zookeeper maintenance in us-east-1
Incident Report for ESS (Public)
Resolved
We have seen no impact from our pre-emptive maintenance.
Posted Jun 25, 2019 - 16:29 UTC
Monitoring
The restart caused no increase in errors to cluster connectivity and freed up the memory on the system. We will monitor for the next 30 minutes and then close out the maintenance. Thank you for your patience!
Posted Jun 25, 2019 - 15:49 UTC
Identified
We have discovered a memory leak in our networking layer for one of our zookeeper coordinating hosts, which requires a restart to clear. This unscheduled maintenance is necessary to avoid a larger impact should the memory leak persist.

This may cause a small spike in 5xx reconnect errors when connecting to clusters in us-east-1.

We are starting this process at 1530 UTC and will be completed by 1630 UTC.
Posted Jun 25, 2019 - 15:31 UTC
This incident affected: AWS N. Virginia (us-east-1) (Elasticsearch connectivity: AWS us-east-1, Kibana connectivity: AWS us-east-1, APM connectivity: AWS us-east-1).