Investigating delayed scheduled snapshots on some deployments

Incident Report for ESS (Public)

Resolved

We have determined that only a very small number of clusters are affected by this issue, and we're looking at them on a case-by-case basis. If your cluster reports that snapshots are out-of-date, please raise a support request and we will look into it. In tandem, we are working on a fix that will be pushed to production ASAP.

Posted Oct 26, 2018 - 04:27 UTC

Update

We are continuing work on a fix for this issue.

Posted Oct 26, 2018 - 00:27 UTC

Identified

We have identified the problem: purging of old snapshots blocks creation of new snapshots. A permanent software fix is being worked on in parallel, and we are verifying a possible immediate fix to alleviate the issue until the software fix is ready and deployed.

Posted Oct 25, 2018 - 22:58 UTC

Investigating

Scheduled snapshots for some clusters are currently delayed.

Posted Oct 25, 2018 - 22:03 UTC