We are running Elasticsearch solution provided by Bitnami in Google Cloud. Cluster runs 4 nodes, 1 master and 3 data nodes.
On Oct 17, Cluster turned Yellow. This is staging cluster so we're not doomed.
Looking at cluster health, we learned two data nodes left the cluster on Oct 17 and they are now in the cluster again. There are 12 unassigned replica shards. Rerouting allocation is failing due to the following error
[NO(target node version [5.5.0] is older than the source node version [5.5.1])]
One node, #4 is running 5.5.1 and all 3 others are 5.5.0. Reading this issue, looks like we have to upgrade Elasticsearch. Followed this guide to upgrade master node to 5.5.1, Elasticsearch didn't start at the end. Rolling back, Elasticsearch restarted successfully.
Is there a robust way of doing this? Worst come worst, we can just spun up a new cluster and restore from backup but this could happen in future and spinning up new cluster every time isn't what we're looking forward to.