We’d have a suggestion.
We currently use the redis chart to spawn a redis sentinel cluster on Kubernetes.
Sentinels are booted as a container in the redis pods. If the pod is deleted, the sentinel goes away with the redis.
As a consequence, when multiple redis pods are deleted, there are cases where sentinels are unable to perform a failover because the majority of votes can’t be reached.
In our particular example, we have 3 replicas and a quorum of 2. The pods run on preemptible nodes, that are cheaper and can be killed by the K8s engine more frequently than primary nodes. If 2 pods are not available at the same time, the system is locked until another pod is available.
This is because the sentinels go away with the pods but, if the sentinels had a different lifecycle, they could still elect the remaining redis replica to become the master.
It would be great if the sentinels were spawned on different pods, so that their lifecycle was decoupled from that of the redis.
Redis sentinels would act as overseers of the system, switching from one redis replica to another as they are killed and respawned by the Kubernetes engine.
Redis sentinels have very little CPU/memory consumption, so they should be more stable and run on any spare room on nodes. The redis are the ones doing the actual job and demanding resources. This would allow for a scenario where redises are spawned on preemptible nodes and sentinels on primary nodes.
The redises can live on preemptible nodes, being deleted anytime, and the sentinels would switch from one to the other.
The sentinels would be more stable on primary nodes and the system would be more resilient overall.
Thanks in advance.
Don’t hesitate if there is more detail that I can provide