MariaDB-Galera cluster deployment in Kubernetes through Helm Chart - Readiness probe failed

Keywords: MariaDB - Linux - How to - Other
Description:
Hi,

I have started to setup my kubernetes cluster and have a few things running already. I now wanted to use the MariaDB-Galera cluster as described here:
Bitnami GitHub Repository. I changed some of the values in the values.yaml to the following:
persistence:
existingClaim: dbstorage
storageClass: "nfs-storage"
and I added a password for the root user in that file. I did removed a lot of the stuff I don’t need from the file but as these items have a default anyways I did not think of them being removed having any issue.
I am using tag: 10.5.9-debian-10-r52. I can see that the pod is successfully created but get the error
Warning BackOff 9m16s (x238 over 110m) kubelet Back-off restarting failed container Warning Unhealthy 4m20s (x396 over 133m) kubelet Readiness probe failed: mysqladmin: connect to server at 'localhost' failed error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mariadb/tmp/mysql.sock' (2)' Check that mysqld is running and that the socket: '/opt/bitnami/mariadb/tmp/mysql.sock' exists!

When I describe the pod I get the following information:

Obviously, I am using a NFS Server for the storage and the mount is an NFS3 mount. I created a PVC called dbstorage which I call and a provisioner automatically provisions the storage. I also can see that there is data in the directory. The pod created 2 files so at least the provisioning worked.

My question obviously is, what do I have to do to successfully deploy the helm chart to make the DB cluster work?

Anyone any idea?

Thank you very much.

Regards

realshadow

Sorry I have to post the description of the pod separately due to the rule that new users can only post 2 links, which I didn’t even try. Anyways the description of the pod is the following:

Namespace:    default
Priority:     0
Node:         k8sworker1/192.168.1.191
Start Time:   Sat, 01 May 2021 17:20:12 -0600
Labels:       app.kubernetes .io/instance=mariadb-galera-cluster
              app.kubernetes .io/name=mariadb-galera
              controller-revision-hash=mariadb-galera-cluster-59c5bcd666
              statefulset.kubernetes .io/pod-name=mariadb-galera-cluster-0
Annotations:  cni.projectcalico .org/podIP: 10.244.230.208/32
              cni.projectcalico .org/podIPs: 10.244.230.208/32
Status:       Running
IP:           10.244.230.208
IPs:
  IP:           10.244.230.208
Controlled By:  StatefulSet/mariadb-galera-cluster
Containers:
  mariadb-galera:
    Container ID:  containerd://165b4cd02833793a5cf62ee8177ff90b633001fab119ecb0fbba32533e4e48bf
    Image:         docker. io/bitnami/mariadb-galera:10.5.9-debian-10-r52
    Image ID:      docker .io/bitnami/mariadb-galera@sha256:ef455467a98606d57e53a4b89f17f647641eff20b6be358ea238653317ed91f7
    Ports:         3306/TCP, 4567/TCP, 4568/TCP, 4444/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      bash
      -ec
      exec /opt/bitnami/scripts/mariadb-galera/entrypoint.sh /opt/bitnami/scripts/mariadb-galera/run.sh

    State:          Running
      Started:      Sat, 01 May 2021 17:32:14 -0600
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Sat, 01 May 2021 17:29:14 -0600
      Finished:     Sat, 01 May 2021 17:32:13 -0600
    Ready:          False
    Restart Count:  4
    Liveness:       exec [bash -ec exec mysqladmin status -u$MARIADB_ROOT_USER -p$MARIADB_ROOT_PASSWORD
] delay=120s timeout=1s period=10s #success=1 #failure=3
    Readiness:  exec [bash -ec exec mysqladmin status -u$MARIADB_ROOT_USER -p$MARIADB_ROOT_PASSWORD
] delay=30s timeout=1s period=10s #success=1 #failure=3
    Environment:
      MY_POD_NAME:                          mariadb-galera-cluster-0 (v1:metadata.name)
      BITNAMI_DEBUG:                        false
      MARIADB_GALERA_CLUSTER_NAME:          galera
      MARIADB_GALERA_CLUSTER_ADDRESS:       gcomm://mariadb-galera-cluster-headless.default.svc.cluster.local
      MARIADB_ROOT_USER:                    root
      MARIADB_ROOT_PASSWORD:                <set to the key 'mariadb-root-password' in secret 'mariadb-galera-cluster'>  Optional: false
      MARIADB_DATABASE:                     my_database
      MARIADB_GALERA_MARIABACKUP_USER:      mariabackup
      MARIADB_GALERA_MARIABACKUP_PASSWORD:  <set to the key 'mariadb-galera-mariabackup-password' in secret 'mariadb-galera-cluster'>  Optional: false
      MARIADB_ENABLE_LDAP:                  no
      MARIADB_ENABLE_TLS:                   no
    Mounts:
      /bitnami/mariadb from data (rw)
      /opt/bitnami/mariadb/.bootstrap from previous-boot (rw)
      /opt/bitnami/mariadb/conf/my.cnf from mariadb-galera-config (rw,path="my.cnf")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kkhs2 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  previous-boot:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  mariadb-galera-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mariadb-galera-cluster-configuration
    Optional:  false
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  dbstorage
    ReadOnly:   false
  kube-api-access-kkhs2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  12m                  default-scheduler  Successfully assigned default/mariadb-galera-cluster-0 to k8sworker1
  Normal   Killing    10m                  kubelet            Container mariadb-galera failed liveness probe, will be restarted
  Normal   Pulled     9m40s (x2 over 12m)  kubelet            Container image "docker.io/bitnami/mariadb-galera:10.5.9-debian-10-r52" already present on machine
  Normal   Created    9m40s (x2 over 12m)  kubelet            Created container mariadb-galera
  Normal   Started    9m39s (x2 over 12m)  kubelet            Started container mariadb-galera
  Warning  Unhealthy  7m30s (x4 over 10m)  kubelet            Liveness probe failed: mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mariadb/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/opt/bitnami/mariadb/tmp/mysql.sock' exists!
  Warning  Unhealthy  2m31s (x49 over 12m)  kubelet  Readiness probe failed: mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mariadb/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/opt/bitnami/mariadb/tmp/mysql.sock' exists!```

Hi @realshadowhunter,

We provide technical support for our containers and charts in our Github repositories. Our team is actively monitoring them and will be glad to help you there, usually within a business day, and the whole community will benefit from your contribution. Could you please post your question by creating a new issue in the bitnami/charts repo?

https://github.com/bitnami/charts/issues

Thanks for your understanding,
Francisco de Paz

@fdepaz appreciate your feedback. I posted my issue on Github [bitnami/mariadb-galera] k8s deployment - Readiness probe failed