MySQL won’t start, error mysqld got signal 6

Keywords: WordPress - Installers - Technical issue - Services (Apache, MariaDB, MySQL…)

bnsupport ID: b8b50864-c6d5-a3e7-1ffc-9521f7852f3c

Description:
Today, Mysql stops working. I do not manage to restart Mysql. I’ve checked hard drive space and it seems that I still have free space available.

[Mysql]

sudo /opt/bitnami/ctlscript.sh status
php-fpm already running
apache already running
mysql not running

Here is what I found in mysql.log :

InnoDB: End of page dump
InnoDB: Page may be a transaction system page
2021-09-11T14:07:08.947191Z 1 [ERROR] [MY-011899] [InnoDB] [FATAL] Unable to read page [page id: space=0, page number=5] into the buffer pool after 100 attempts. The most probable cause of this error may be that the table has been corrupted. Or, the table was compressed with with an algorithm that is not supported by this instance. If it is not a decompress failure, you can try to fix this problem by using innodb_force_recovery. Please see http://dev.mysql.com/doc/refman/8.0/en/ for more details. Aborting…
2021-09-11T14:07:08.948265Z 1 [ERROR] [MY-013183] [InnoDB] Assertion failure: ut0ut.cc:552 thread 140657203504896
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
14:07:08 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x6788200
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 7fed4ea65ce0 thread_stack 0x46000
/opt/bitnami/mysql/bin/mysqld.bin(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x1edf6fe]
/opt/bitnami/mysql/bin/mysqld.bin(handle_fatal_signal+0x323) [0xfa2ef3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fed5bab63c0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7fed5a7b618b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7fed5a795859]
/opt/bitnami/mysql/bin/mysqld.bin() [0xd25c6e]
/opt/bitnami/mysql/bin/mysqld.bin(ib::fatal::~fatal()+0x95) [0x2199315]
/opt/bitnami/mysql/bin/mysqld.bin(Buf_fetch<Buf_fetch_normal>::read_page()+0x189) [0x21f67e9]
/opt/bitnami/mysql/bin/mysqld.bin(Buf_fetch_normal::get(buf_block_t*&)+0x18) [0x21f6838]
/opt/bitnami/mysql/bin/mysqld.bin(Buf_fetch<Buf_fetch_normal>::single_page()+0x43) [0x21fade3]
/opt/bitnami/mysql/bin/mysqld.bin(buf_page_get_gen(page_id_t const&, page_size_t const&, unsigned long, buf_block_t*, Page_fetch, char const*, unsigned long, mtr_t*, bool)+0x19a) [0x21fdc9a]
/opt/bitnami/mysql/bin/mysqld.bin(Double_write::get(mtr_t*)+0x4a) [0x220446a]
/opt/bitnami/mysql/bin/mysqld.bin(Double_write::init_v1(unsigned int&, unsigned int&)+0xf3) [0x22060b3]
/opt/bitnami/mysql/bin/mysqld.bin(dblwr::v1::init()+0x17) [0x22061f7]
/opt/bitnami/mysql/bin/mysqld.bin(srv_start(bool)+0x1a78) [0x2144578]
/opt/bitnami/mysql/bin/mysqld.bin() [0x1fb98b7]
/opt/bitnami/mysql/bin/mysqld.bin(dd::bootstrap::DDSE_dict_init(THD*, dict_init_mode_t, unsigned int)+0x81) [0x1c00ec1]
/opt/bitnami/mysql/bin/mysqld.bin(dd::upgrade_57::do_pre_checks_and_initialize_dd(THD*)+0x18b) [0x1eaad8b]
/opt/bitnami/mysql/bin/mysqld.bin() [0x1058098]
/opt/bitnami/mysql/bin/mysqld.bin() [0x23e2abc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fed5baaa609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fed5a892293]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

Found recent error messages in the MySQL error log:

Most likely, you have hit a bug, but this error can also be caused by
malfunctioning hardware.

Hi @ExoGeek,

There seem to be a lot of requests coming from the same few IP addresses:

sudo tail -n 100000 /opt/bitnami/mysql/logs/access_log | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10 | awk '{print $1}' 

This can indicate a bot attack. Regarding MySQL, it could be that MySQL was busy writing data to disk, but couldn’t finish because it crashed. (Maybe because of a bot attack?) and now the table is corrupted. Do you have a data backup? In that case the best option is to restore the data.

Regards,
Michiel

Hello @michiel,

Thanks for your response and thanks for the explanation…

I’ve restored the server with the snapshot of the previous night (before the hack). Everything seems to be ok right now.

Is there anything I can do to avoid such issues in the future ? For example : limit the number of incoming mysql requests ?

Thanks so much for your support.
ExoGeek

Hi @ExoGeek,

I recommend using security plugins like WordFence or Sucuri to prevent your site from being hacked. MySQL is able to handle quite some requests in case of normal usage. If with normal (non hack) traffic your instance can’t keep up, I’d recommend upgrading to a more powerful instance instead of limiting MySQL requests.

Regards,
Michiel