Elastic Cloud Status

All Systems Operational

Cluster Management Operational

Cluster Management Console Service Operational

Cluster Management API Operational

Cluster Orchestration Operational

Cluster Metrics Operational

Cluster Snapshots Operational

AWS Marketplace Operational

GCP us-central1 Operational

Cluster Connectivity: GCP us-central1 Operational

Kibana Connectivity: GCP us-central1 Operational

APM Connectivity: GCP us-central1 Operational

GCP us-west1 Operational

Cluster Connectivity: GCP us-west1 Operational

Kibana Connectivity: GCP us-west1 Operational

APM Connectivity: GCP us-west1 Operational

GCP europe-west1 Operational

Cluster Connectivity: GCP europe-west1 Operational

Kibana Connectivity: GCP europe-west1 Operational

APM Connectivity: GCP europe-west1 Operational

GCP europe-west3 Operational

Cluster Connectivity: GCP europe-west3 Operational

Kibana Connectivity: GCP europe-west3 Operational

APM Connectivity: GCP europe-west3 Operational

Google Cloud Platform Operational

Google Compute Engine Operational

Google Cloud Storage Operational

AWS N. Virginia (us-east-1) Operational

Cluster Connectivity: AWS us-east-1 Operational

AWS EC2 Health: us-east-1 Operational

Snapshot Storage Infrastructure (S3): us-east-1 Operational

Kibana Connectivity: AWS us-east-1 Operational

APM Connectivity: AWS us-east-1 Operational

AWS N. California (us-west-1) Operational

Cluster Connectivity: AWS us-west-1 Operational

AWS EC2 Health: us-west-1 Operational

Snapshot Storage Infrastructure (S3): us-west-1 Operational

Kibana Connectivity: AWS us-west-1 Operational

APM Connectivity: AWS us-west-1 Operational

AWS Ireland (eu-west-1) Operational

Cluster Connectivity: AWS eu-west-1 Operational

AWS EC2 Health: eu-west-1 Operational

Snapshot Storage Infrastructure (S3): eu-west-1 Operational

Kibana Connectivity: AWS eu-west-1 Operational

APM Connectivity: AWS eu-west-1 Operational

AWS Frankfurt (eu-central-1) Operational

Cluster Connectivity: AWS eu-central-1 Operational

AWS EC2 Health: eu-central-1 Operational

Snapshot Storage Infrastructure (S3): eu-central-1 Operational

Kibana Connectivity: AWS eu-central-1 Operational

APM Connectivity: AWS eu-central-1 Operational

AWS Oregon (us-west-2) Operational

Cluster Connectivity: AWS us-west-2 Operational

AWS EC2 Health: us-west-2 Operational

Snapshot Storage Infrastructure (S3): us-west-2 Operational

Kibana Connectivity: AWS us-west-2 Operational

APM Connectivity: AWS us-west-2 Operational

AWS São Paulo (sa-east-1) Operational

Cluster Connectivity: AWS sa-east-1 Operational

AWS EC2 Health: sa-east-1 Operational

Snapshot Storage Infrastructure (S3): sa-east-1 Operational

Kibana Connectivity: AWS sa-east-1 Operational

APM Connectivity: AWS sa-east-1 Operational

AWS Singapore (ap-southeast-1) Operational

Cluster Connectivity: AWS ap-southeast-1 Operational

AWS EC2 Health: ap-southeast-1 Operational

Snapshot Storage Infrastructure (S3): ap-southeast-1 Operational

Kibana Connectivity: AWS ap-southeast-1 Operational

APM Connectivity: AWS ap-southeast-1 Operational

AWS Sydney (ap-southeast-2) Operational

Cluster Connectivity: AWS ap-southeast-2 Operational

AWS EC2 Health: ap-southeast-2 Operational

Snapshot Storage Infrastructure (S3): ap-southeast-2 Operational

Kibana Connectivity: AWS ap-southeast-2 Operational

APM Connectivity: AWS ap-southeast-2 Operational

AWS Tokyo (ap-northeast-1) Operational

Cluster Connectivity: AWS ap-northeast-1 Operational

AWS EC2 Health: ap-northeast-1 Operational

Snapshot Storage Infrastructure (S3): ap-northeast-1 Operational

Kibana Connectivity: AWS ap-northeast-1 Operational

APM Connectivity: AWS ap-northeast-1 Operational

Heroku Operational

Elastic Maps Service Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Past Incidents

Jun 20, 2019

Cluster Logs and Metrics for customer clusters is currently delayed

Resolved - This incident has been resolved.
Jun 20, 03:54 UTC

Update - Cluster logging and metric data should now be up to date for all deployments. We'll continue to monitor our ingestion pipelines and monitoring clusters as per our normal monitoring.

There should be no ongoing customer impact as part of this incident.
Jun 20, 03:28 UTC

Monitoring - Cluster logging and metric data for most clusters should now be up to date. Deployments in AWS eu-central-1, GCP us-central1 and GCP europe-west1 regions will still be delayed, however those regions are processing queues and will be up to date within the hour.

We will update this issue within the next 30 minutes.
Jun 20, 03:01 UTC

Identified - Customer logs and metrics appear to be flowing again, with queues trending downwards. We expect cluster logging and metric data to be up to date within the next hour.

One of our backend monitoring clusters had entered a split-brain state which was impacting indexing rates within our Logstash consumers. We're still investigating what caused this cluster to get into this state, but believe two nodes were impacted by our recent hot-warm incident resulting in those nodes not able to keep up with cluster state changes. We've restored this cluster to an operational state, which in turn restored our log ingestion rates.

We'll post another update in 30 minutes, or as new information comes to hand.
Jun 20, 02:43 UTC

Investigating - Currently cluster logs and metrics across all regions are delayed. At this stage you won't be able to see up to date logs or metrics for your clusters. You can still access historical logs and metrics. We're investigating the root cause and will post an update shortly.
Jun 20, 02:03 UTC

Jun 19, 2019

No incidents reported.

Jun 18, 2019

No incidents reported.

Jun 17, 2019

No incidents reported.

Jun 16, 2019

No incidents reported.

Jun 15, 2019

No incidents reported.

Jun 14, 2019

No incidents reported.

Jun 13, 2019

No incidents reported.

Jun 12, 2019

No incidents reported.

Jun 11, 2019

No incidents reported.

Jun 10, 2019

Increased failure rate in configuration changes of hot-warm deployments

Resolved - We have tested extensively over the weekend and monitored our failure rates. The rate of failures has dropped to our usual level and we are now resolving this incident. Thank you for your patience!
Jun 10, 15:02 UTC

Monitoring - We have completed the rollout to all production versions and confirmed that hot-warm deployment changes are now succeeding. We will monitor for the next day.
Jun 8, 00:28 UTC

Update - We have tested the fix in our staging environment and are working on getting it rolled out to production. We have confirmed that the rate of errors on deployment changes for hot-warm has decreased to normal levels. The production rollout is ongoing, and we will update you when it is completed.
Jun 7, 22:26 UTC

Identified - Identified:

This incident is impacting all hot-warm cluster deployment changes in AWS regions. Connectivity to clusters is not impacted.

The preliminary mitigation that we rolled out last evening was a timeout increase, which allowed more hot-warm plans to succeed. This gave us a little bit of breathing room to dig in deep into the code base and spend some time profiling thread contention, and we've had a breakthrough. We discovered that a networking method was writing a state file to disk with every run, rather than when it changed, causing an increased load on disk and additional thread contention. We have merged a fix and have begun the testing and rollout process.

We will update you in six hours with our progress.
Jun 7, 16:48 UTC

Update - Based on preliminary findings the team was able to conclude recent configuration change has improved the success rate of new deployment creations. We are still in process of determining the root cause of increased failure rate in plan configuration changes of hot-warm deployments.

We will provide an update within the next 6 hours.
Jun 7, 09:50 UTC

Update - We've deployed our interim mitigations which increase the thresholds causing hot-warm plan changes to fail. We're awaiting data on the effectiveness of these changes, but expect these to reduce the failure rates of hot-warm deployment changes.

We'll update this issue within the next 4 hours as data comes to hand.
Jun 7, 06:24 UTC

Investigating - We are observing an elevated error rate in Deployment changes for Elasticsearch clusters in a hot-warm setup.

Customers making changes to or creating new hot-warm deployments may be experiencing errors, shown in the Deployment Activity tab.

We have identified that containers holding the Elasticsearch node processes are taking longer than expected to start, exceeding a threshold that leads to marking deployment changes as failed.

A team of cloud engineers are working on an interim mitigation that addresses the effects of this issue, and we expect a decrease in failure rates soon.

A new update will be posted within 4 hours.
Jun 6, 19:23 UTC

Jun 9, 2019

No incidents reported.

Jun 7, 2019

Kibana/APM network connectivity issues in AWS eu-central-1

Resolved - Based on the data gathered over the past hour we can conclude the connectivity between Kibana/APM and Elasticsearch clusters has been successfully restored. Customers should not experience any issues with connectivity to Kibana instances.

What happened: The Elastic Cloud Proxy tier is currently at the end of an implementation phase which has introduced a new Cloud Proxy for handling client requests. Part of the implementation phase involved using haproxy to conditionally route traffic between the old and new proxies so that we could introduce the change gradually and with least impact.
As part of the work to remove haproxy from this stack a configuration change was made to the AWS eu-central-1 region which resulted in inter-deployment (Kibana -> Elasticsearch; APM -> Elasticsearch) communication issues.

This change has now been reverted.
Jun 7, 13:03 UTC

Monitoring - Problematic proxy release has been successfully rolled back. Connectivity to Kibana instances should now be restored. We are currently monitoring the situation and will provide more details about the incident within the next 60 minutes.
Jun 7, 11:12 UTC

Update - Latest proxy release in AWS eu-central-1 region has caused issues with network connectivity from Kibana/APM to Elasticsearch. The release is being rolled back now. Customers might experience issues connecting to their Kibana instances. Functionality of APM might be affected.

Direct access to Elasticsearch clusters is not impacted.

We will provide status update within the next 60 minutes.
Jun 7, 11:02 UTC

Investigating - We are currently investigating this issue.
Jun 7, 10:47 UTC

Jun 6, 2019

Failures in API Console

Resolved - All affected components have been re-deployed with the correct versions. This incident has been resolved.
Jun 6, 06:44 UTC

Monitoring - The deployment of one of our API services has completed and users should no longer be experiencing any issues when interacting with deployments via the console. Executing requests via the API Console, and accessing deployment snapshots should be succeed.

We're still in the process of re-deploying the second API service, however there should be no further user impact whilst this completes.
Jun 6, 06:15 UTC

Update - We've reviewed the versions of other related components and have found another service which requires re-deployment. We're working on re-deploying this second component with the correct version, however this will delay the resolution of this incident.

We expect both components to be fully deployed within the next 45 minutes, and will update this incident at that time or as any changes come to hand.
Jun 6, 05:58 UTC

Identified - We've identified a version mismatch in some recently deployed components. We believe this to be causing the failures in executing requests via the API Console.

We're currently working on redeploying this component with the correct version, and anticipate this will be complete within the next 15 minutes.

We'll update this issue when this deployment is complete.
Jun 6, 05:38 UTC

Investigating - We're encountering issues executing requests against a deployment via the customer console. Users may experience errors when executing requests via the 'API Console' or when viewing, or restoring snapshots for a deployment.

There are no general cluster connectivity issues.

We're currently investigating this issue and will update this status page within the next hour, or as more information comes to hand.
Jun 6, 05:26 UTC