(!) MySQL Server not Starting

Keywords: MySQL - AWS - Technical issue - Other
Description:
Hello team,

Situation:
Something changed on the settings of our server. If we create a snapshot 1 week ago everything is fine (apache/mysql… is working). But if we create a new instance from a actually made snapshot the mysql server is not starting.

What we checked:

We are not running out of space. A upgrade of disk space to a lager instance brings no luck:

root@ip:/home/bitnami# sudo du -ch --max-depth=1 /opt/bitnami
52K/opt/bitnami/config
32M/opt/bitnami/letsencrypt
116K/opt/bitnami/scripts
2.1G/opt/bitnami/apache2
3.0M/opt/bitnami/stats
26M/opt/bitnami/bncert
30M/opt/bitnami/bnsupport
91G/opt/bitnami/apps
4.2M/opt/bitnami/bnhelper
3.6M/opt/bitnami/varnish
3.5M/opt/bitnami/sqlite
81M/opt/bitnami/php
9.9G/opt/bitnami/mysql
1.6M/opt/bitnami/licenses
8.0K/opt/bitnami/img
104K/opt/bitnami/var
145M/opt/bitnami/common
103G/opt/bitnami
103Gtotal

(320 GB SSD lightsail)

There is no other server blocking us:

root@:/opt/bitnami# ps aux | grep 'apache\|mysql\|nginx\|/opt/bitnami'
root      2587  0.0  0.0 303028 10700 ?        Ss   16:13   0:00 php-fpm: master process (
/opt/bitnami/php/etc/php-fpm.conf)
root      2603  0.0  0.3 207096 56256 ?        Ss   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2645  0.0  0.0 815704 14260 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2646  0.0  0.0 815704 14272 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2647  0.0  0.0 815704 14332 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2648  0.0  0.0 815704 14268 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2649  0.0  0.0 815704 14268 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2650  0.0  0.0 815704 14340 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2651  0.0  0.1 1208920 18384 ?       Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2652  0.0  0.0 815704 14272 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2653  0.0  0.0 815704 14336 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2654  0.0  0.0 815704 14272 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2655  0.0  0.1 1077848 17276 ?       Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2656  0.0  0.1 1143384 17368 ?       Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2657  0.0  0.1 946776 17852 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2658  0.0  0.0 815704 14272 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2659  0.0  0.0 815704 14356 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2660  0.0  0.0 815704 14296 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
root      6127  0.0  0.0  14352   968 pts/0    S+   16:25   0:00 grep --color=auto apache\
|mysql\|nginx\|/opt/bitnami

Memory is free:

  total        used        free      shared  buff/cache   available
Mem: 15G 225M 12G 56M 2.6G 15G
Swap: 0B 0B 0B

sudo systemctl status bitnami.service:

Loaded: loaded (/etc/init.d/bitnami; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-06-22 16:13:23 UTC; 46min ago
Process: 1312 ExecStart=/etc/init.d/bitnami start (code=exited, status=3)
Tasks: 1079
Memory: 212.7M
CPU: 12.114s
CGroup: /system.slice/bitnami.service
├─2587 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf)
├─2588 php-fpm: pool wordpress
├─2589 php-fpm: pool wordpress
├─2590 php-fpm: pool wordpress
├─2591 php-fpm: pool wordpress
├─2592 php-fpm: pool wordpress
├─2593 php-fpm: pool wordpress
├─2603 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2645 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2646 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2647 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2648 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2649 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2650 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2651 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2652 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2653 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2654 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2655 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2656 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2657 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2658 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2659 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2660 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
└─3759 /usr/bin/gonit

Mysql Logs:
empty

System logs:
Jun 22 08:58:07 : /opt/bitnami/mysql/scripts/ctl.sh : mysql could not be started
Jun 22 08:58:18 : /opt/bitnami/php/scripts/ctl.sh : php-fpm started
Jun 22 08:58:23 : Syntax OK

We think of a something like a permission issue?

/opt/bitnami/mysql# l -lh
drwxr-xr-x  2 root    root 4.0K Feb 19  2020 bin/
drwxr-xr-x  2 bitnami root 4.0K Apr  7  2020 bitnami/
-r-xr--r--  1 root    root 8.2M Feb 19  2020 bnconfig*
drwxr-x---  7 mysql   root 4.0K Jun 22 18:13 data/
drwxr-xr-x  3 root    root 4.0K Feb 19  2020 include/
drwxr-xr-x  7 root    root 4.0K Feb 19  2020 lib/
-rw-r--r--  1 root    root 400K Sep 20  2019 LICENSE
-rw-r--r--  1 root    root 101K Sep 20  2019 LICENSE.router
-rw-r--r--  1 bitnami root  674 Jun 22 17:53 my.cnf
-rw-r--r--  1 root    root  687 Sep 20  2019 README
-rw-r--r--  1 root    root  700 Sep 20  2019 README.router
drwxrwxr-x  2 root    root 4.0K Feb 19  2020 run/
drwxr-xr-x  2 root    root 4.0K Feb 19  2020 scripts/
drwxr-xr-x 28 root    root 4.0K Feb 19  2020 share/
drwxr-xr-x  2 root    root 4.0K Feb 19  2020 support-files/
drwxrwxrwx  2 root    root 4.0K Jun 21 22:26 tmp/
drwxr-xr-x  3 root    root 4.0K Feb 19  2020 var/

Bitnami Support Ticket:
Cant be uploaded. How do I upload it manually?

Hi @eSpox,

I just downloaded the Bitnami Support tool file you shared in the private thread and took a look at it. I can see that you didn’t change the MySQL configuration file (/opt/bitnami/mysql/my.cnf) but the tool didn’t get the /opt/bitnami/mysql/bitnami/my.cnf file, can you post its content here?

sudo cat /opt/bitnami/mysql/bitnami/my.cnf

I also checked that the permissions of the MySQL folder seem correct, can you try to start the MySQL database manually?

MYSQL_PIDFILE=/opt/bitnami/mysql/data/mysqld.pid

MYSQL_START="/opt/bitnami/mysql/bin/mysqld_safe --defaults-file=/opt/bitnami/mysql/my.cnf --mysqld=mysqld.bin --socket=/opt/bitnami/mysql/tmp/mysql.sock --old-passwords --datadir=/opt/bitnami/mysql/data --log-error=/opt/bitnami/mysql/data/mysqld.log  --pid-file=$MYSQL_PID
FILE --lower-case-table-names=1 --default-table-type=InnoDB"

If you do not get any error when running the command, is there any error in the log now?

sudo tail -n 20 /opt/bitnami/mysql/data/mysqld.log

Hey!

We was hardly struggling because there was no errors or warnings at all. But after reset the permissions of “opt/mysql/data”, the mysql can be started and everything working fine.

So I would like to try going back to the base problem we had: If we tweak the my.cnf, our mysql is not starting again. If we add this settings, the mysql can not be started:

[mysqladmin]
user=root

[mysqld]
basedir=/opt/bitnami/mysql
datadir="/opt/bitnami/mysql/data"
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
tmpdir=/opt/bitnami/mysql/tmp
max_allowed_packet=32M
bind-address=127.0.0.1
skip-name-resolve=1
expire_logs_days=7
character-set-server=utf8mb4
collation-server=utf8mb4_general_ci
**key_buffer_size= 32M**
**thread_stack=256K**
**thread_cache_size=32**
**tmp_table_size=16M**
**table_cache=128**
**query_cache_limit=4M**
**query_cache_size=32M**

default-authentication-plugin=mysql_native_password
[client]
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock

default-character-set=utf8mb4
[manager]
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
pid-file=/opt/bitnami/mysql/tmp/manager.pid
default-mysqld-path=/opt/bitnami/mysql/bin/mysqld.bin


!include /opt/bitnami/mysql/bitnami/my.cnf

my.cnf of "bitnami/mysql/bitnami:

# 
# Bitnami MySQL Configuration
# Copyright 2020 Bitnami.com All Rights Reserved
# 
# Note: This file will be modified on server size changes
#

[mysqld]
#wait_timeout = 120
long_query_time = 1
innodb_buffer_pool_size=2048M
#innodb_log_file_size=128M
#tmp_table_size=64M
#max_connections = 2500
#max_user_connections = 2500
#innodb_flush_method=O_DIRECT
#key_buffer_size=64M

@jota you are a luck bringer. We made a research and fixed the problem! (y) But maybe you could add abit more context to it:

The “bitnami” my.cnf config do not start when depreacted settings are made. We are using the Mysql 8.X.X and these three values got very big changes from the mysql team:

table cache -> renamed -> table open cache
query cache and query cache limit are depracted and will no longer do anything

We renamed “table cache” and removed the query cache thing and the mysql is starting again.

Is it normal, that the mysql do not start without any warnings because of depreacted lines?

Hi @eSpox,

I’m glad you fixed your issue! If you try to start MySQL with an invalid configuration, an error message should be shown into the MySQL logfile (located at /opt/bitnami/mysql/data/mysqld.log). It is weird you are not getting it though. Did you modify any setting related to that?

Hey,

no we did not modify any ettings related to that. The mysql log is “always” empty.

Hi @eSpox,

We have a Support Tool that will gather relevant information for us to analyze your configuration and logs. Could you please execute it on the machine where the stack is running by following the steps described in the guide below?

Please note that you need to paste the code ID that is shown at the end.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.