Error when starting MySQL service in AWS Bitnami Wordpress Multisite

Keywords: MySQL - AWS - Technical issue - Other
bnsupport ID: b2ba86d2-bc81-0c19-14ad-9e3b52e15f76
Description:
Good afternoon,

I’m having trouble when starting the Bitnami service in my AWS EC2 instance. This problem suddenly happened today in the morning (it was working perfectly yesterday).

Problem

My website shows the message “Error establishing a database connection”. It shows me apache is running, but MySQL isn’t.

My steps

Step 1: restarting the service

When I run sudo service bitnami restart, the following error appears

Job for bitnami.service failed because the control process exited with error code.
See "systemctl status bitnami.service" and "journalctl -xe" for details.

Then, running sudo systemctl status bitnami.service, the following info appears

● bitnami.service - LSB: bitnami init script
   Loaded: loaded (/etc/init.d/bitnami; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2021-05-14 16:56:30 UTC; 21min ago
  Process: 586 ExecStart=/etc/init.d/bitnami start (code=exited, status=3)
    Tasks: 1079 (limit: 4915)
   Memory: 365.8M
   CGroup: /system.slice/bitnami.service
           ├─1171 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf)
           ├─1172 php-fpm: pool www
           ├─1173 php-fpm: pool www
           ├─1174 php-fpm: pool www
           ├─1175 php-fpm: pool www
           ├─1176 php-fpm: pool www
           ├─1177 php-fpm: pool www
           ├─1183 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1222 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1223 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1225 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1226 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1227 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1228 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1229 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1230 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1231 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1232 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1233 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1234 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1235 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1236 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1237 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─1238 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
           ├─2286 /usr/bin/gonit
           ├─6109 /bin/sh /opt/bitnami/mysql/scripts/ctl.sh start
           └─6513 sleep 3

May 14 16:56:30 ip-10-0-70-186 bitnami[586]: ## 2021-05-14 16:56:30+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/030_update_ip...
May 14 16:56:30 ip-10-0-70-186 bitnami[586]: ## 2021-05-14 16:56:30+00:00 ## INFO ## Automatic IP update disabled
May 14 16:56:30 ip-10-0-70-186 bitnami[586]: ## 2021-05-14 16:56:30+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/040_update_welcome_file...
May 14 16:56:30 ip-10-0-70-186 bitnami[586]: ## 2021-05-14 16:56:30+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/050_bitnami_credentials_file...
May 14 16:56:30 ip-10-0-70-186 bitnami[586]: ## 2021-05-14 16:56:30+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/060_start_gonit...
May 14 16:56:30 ip-10-0-70-186 bitnami[586]: Starting gonit daemon
May 14 16:56:30 ip-10-0-70-186 bitnami[586]: ## 2021-05-14 16:56:30+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/070_clean_metadata...
May 14 16:56:30 ip-10-0-70-186 systemd[1]: bitnami.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
May 14 16:56:30 ip-10-0-70-186 systemd[1]: bitnami.service: Failed with result 'exit-code'.
May 14 16:56:30 ip-10-0-70-186 systemd[1]: Failed to start LSB: bitnami init script.

Step 2: printing MySQL’s log file

In this way, I printed the MySQL log file into the terminal (cat /opt/bitnami/mysql/data/mysqld.log) to find out the problem, and those lines appeared:

2021-05-14T13:37:55.846458Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.23) starting as process 10420
2021-05-14T13:37:55.855981Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-05-14T13:37:57.581439Z 1 [Warning] [MY-012637] [InnoDB] 1048576 bytes should have been written. Only 978944 bytes written. Retrying for the remaining bytes.
2021-05-14T13:37:57.581598Z 1 [Warning] [MY-012638] [InnoDB] Retry attempts for writing partial data failed.
2021-05-14T13:37:57.581716Z 1 [ERROR] [MY-012639] [InnoDB] Write to file ./ibtmp1 failed at offset 11534336, 1048576 bytes should have been written, only 978944 were written. Operating system error number 28. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.
2021-05-14T13:37:57.581812Z 1 [ERROR] [MY-012640] [InnoDB] Error number 28 means 'No space left on device'
2021-05-14T13:37:57.582005Z 1 [ERROR] [MY-012267] [InnoDB] Could not set the file size of './ibtmp1'. Probably out of disk space
2021-05-14T13:37:57.582100Z 1 [ERROR] [MY-012926] [InnoDB] Unable to create the shared innodb_temporary.
2021-05-14T13:37:57.582201Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2021-05-14T13:37:57.963649Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2021-05-14T13:37:57.964010Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2021-05-14T13:37:57.964313Z 0 [ERROR] [MY-010119] [Server] Aborting
2021-05-14T13:37:57.964870Z 0 [System] [MY-010910] [Server] /opt/bitnami/mysql/bin/mysqld.bin: Shutdown complete (mysqld 8.0.23)  MySQL Community Server - GPL.

Step 3: checking disk space

So, as Error number 28 appeared meaning that I shouldn’t have enough disk space, I did the command df -h and it showed me this:

udev             7.8G     0  7.8G   0% /dev
tmpfs            1.6G  8.4M  1.6G   1% /run
/dev/nvme0n1p1    50G   24G   24G  51% /
tmpfs            7.8G     0  7.8G   0% /dev/shm
tmpfs            5.0M     0  5.0M   0% /run/lock
tmpfs            7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/nvme0n1p15  124M  278K  124M   1% /boot/efi
tmpfs            1.6G     0  1.6G   0% /run/user/1000

that is far from being full. :’)

Step 4: further exploration

I also watched some youtube videos saying to kill all the processes related to bitnami and restart, but none of them helped me. Also, I did /etc/init.d/bitnami status and it returned me:

php-fpm already running
apache already running
mysql not running

that clearly shows me a problem with MySQL.

Also, I tried removing (actually, renaming) the file /opt/bitnami/mysql/tmp/mysql.sock.lock (since it shows ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/bitnami/mysql/tmp/mysql.sock' (2) when running the command mysql), but no luck so far.

I tried expanding the HD size, clearing my cache files, but also I didn’t have any signal of improvement.

So, after everything…

How may I progress from this point? Is there something else to check or to change to make it back online? Or maybe search for some limitation of my system?

I’m looking forward to your response :slight_smile:

Hello @alexandre.bellas,

I’m not seeing any other red flag besides the out of space error. Could you try restarting your AWS instance and then restarting MySQL? It may be that MySQL is not detecting correctly the disk size.

After that, please share the output of the following command if the service still doesn’t start:

sudo tail -20 /opt/bitnami/mysql/data/mysqld.log

Regards,
Francisco de Paz

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.