We identified that between Feb 12th at 21:30 UTC and Feb 18 at 10:00 UTC some network related settings were not being properly applied on newly created deployment nodes in AWS. We determined that this impacted 2-3% of AWS deployments, specifically the following situations:
1. Newly created deployments. 2. Deployments that had topology changes resulting in node creation during the window.
The impact would have been observed by nodes occasionally dropping and returning to their respective clusters (node left events). The root cause was found to be a failure in settings application and has been remediated, we are also working on improving how we monitor these configuration settings to prevent such a situation in the future.
We would like to apologize for the inconvenience and if you still see issues please reach out to through https://support.elastic.co and we will be happy to assist.
Posted Feb 18, 2020 - 17:31 UTC
This incident affected: AWS Tokyo (ap-northeast-1) (Deployment hosts: AWS ap-northeast-1), AWS São Paulo (sa-east-1) (Deployment hosts: AWS sa-east-1), AWS N. Virginia (us-east-1) (Deployment hosts: AWS us-east-1), AWS Frankfurt (eu-central-1) (Deployment hosts: AWS eu-central-1), AWS Ireland (eu-west-1) (Deployment hosts: AWS eu-west-1), AWS London (eu-west-2) (Deployment hosts: AWS eu-west-2), AWS Singapore (ap-southeast-1) (Deployment hosts: AWS ap-southeast-1), AWS Oregon (us-west-2) (Deployment hosts: AWS us-west-2), AWS N. California (us-west-1) (Deployment hosts: AWS us-west-1), and AWS Sydney (ap-southeast-2) (Deployment hosts: AWS ap-southeast-2).