MySQL stopped working

:warning: IMPORTANT, please fill the questions

We assume you are using Bitnami to deploy your application.

  • Which version of the application are you using?:
    AWS LightSail

  • Please choose how you got the application: Installer (Windows, Linux, macOS), cloud image (AWS, GCE, Azure, …) or VM (VMDK, VBOX):
    AWS

  • Have you installed any plugin or modified any configuration file?:

  • Describe here your question/suggestion/issue (expected and actual results):
    MySQL stopped working suddenly and getting Error establishing a database connection. Tried to reboot the server and also added an additional disk after seeing the disk space issue.

Following is Mysqld.log

bitnami@ip-172-26-2-191:~$ sudo tail -n 100 /opt/bitnami/mysql/data/mysqld.log
2020-11-16T04:36:15.927023Z 82483 [ERROR] [MY-011072] [Server] Binary logging not possible. Message: An error occurred during flush stage of the commit. 'binlog_error_action' is set to 'ABORT_SERVER'. Hence aborting the server..
04:36:15 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x490ea20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fab885f7d80 thread_stack 0x46000
/opt/bitnami/mysql/bin/mysqld.bin(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x1ed6cce]
/opt/bitnami/mysql/bin/mysqld.bin(handle_fatal_signal+0x323) [0xfb2d23]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7fabcd30d390]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7fabcb34c438]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7fabcb34e03a]
/opt/bitnami/mysql/bin/mysqld.bin() [0xce2179]
/opt/bitnami/mysql/bin/mysqld.bin(MYSQL_BIN_LOG::handle_binlog_flush_or_sync_error(THD*, bool)+0x2ad) [0x1b24bad]
/opt/bitnami/mysql/bin/mysqld.bin(MYSQL_BIN_LOG::ordered_commit(THD*, bool, bool)+0x108) [0x1b33c18]
/opt/bitnami/mysql/bin/mysqld.bin(MYSQL_BIN_LOG::commit(THD*, bool)+0x381) [0x1b34f31]
/opt/bitnami/mysql/bin/mysqld.bin(ha_commit_trans(THD*, bool, bool)+0x39d) [0x10bb78d]
/opt/bitnami/mysql/bin/mysqld.bin(trans_commit_stmt(THD*, bool)+0x2d) [0xf76f8d]
/opt/bitnami/mysql/bin/mysqld.bin(mysql_execute_command(THD*, bool)+0x3f29) [0xe8cfa9]
/opt/bitnami/mysql/bin/mysqld.bin(mysql_parse(THD*, Parser_state*)+0x327) [0xe8e8b7]
/opt/bitnami/mysql/bin/mysqld.bin(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x2474) [0xe91244]
/opt/bitnami/mysql/bin/mysqld.bin(do_command(THD*)+0x174) [0xe91cb4]
/opt/bitnami/mysql/bin/mysqld.bin() [0xfa4c78]
/opt/bitnami/mysql/bin/mysqld.bin() [0x23b04ac]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fabcd3036ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fabcb41e4dd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (49a2d58): UPDATE `proip_options` SET `option_value` = 'a:33:{i:1605403839;a:1:{s:26:\"action_scheduler_run_queue\";a:1:{s:32:\"0d04ed39571b55704c122d726248bbac\";a:3:{s:8:\"schedule\";s:12:\"every_minute\";s:4:\"args\";a:1:{i:0;s:7:\"WP Cron\";}s:8:\"interval\";i:60;}}}i:1605403899;a:1:{s:26:\"action_scheduler_run_queue\";a:1:{s:32:\"0d04ed39571b55704c122d726248bbac\";a:3:{s:8:\"schedule\";s:12:\"every_minute\";s:4:\"args\";a:1:{i:0;s:7:\"WP Cron\";}s:8:\"interval\";i:60;}}}i:1605404488;a:1:{s:24:\"aiowps_hourly_cron_event\";a:1:{s:32:\"40cd750bba9870f18aada2478b24840a\";a:3:{s:8:\"schedule\";s:6:\"hourly\";s:4:\"args\";a:0:{}s:8:\"interval\";i:3600;}}}i:1605406165;a:1:{s:34:\"wp_privacy_delete_old_export_files\";a:1:{s:32:\"40cd750bba9870f18aada2478b24840a\";a:3:{s:8:\"schedule\";s:6:\"hourly\";s:4:\"args\";a:0:{}s:8:\"interval\";i:3600;}}}i:1605416546;a:1:{s:33:\"heatmap_for_wp_cron_check_account\";a:1:{s:32:\"40cd750bba9870f18aada2478b24840a\";a:3:{s:8:\"schedule\";s:5:\"daily\";s:4:\"args\";a:0:{}s:8:\"interval\";i
Connection ID (thread ID): 82483
Status: KILL_CONNECTION

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2020-11-16T04:36:16.433635Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T04:36:16.434005Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 10417
2020-11-16T04:36:17.273996Z 1 [Warning] [MY-012637] [InnoDB] 81920 bytes should have been written. Only 77824 bytes written. Retrying for the remaining bytes.
2020-11-16T04:36:17.274182Z 1 [Warning] [MY-012638] [InnoDB] Retry attempts for writing2020-11-16T04:36:17.274846Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Out of disk space.
2020-11-16T04:36:17.776561Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2020-11-16T04:36:17.776877Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2020-11-16T04:36:17.777179Z 0 [ERROR] [MY-010119] [Server] Aborting
2020-11-16T04:36:17.784384Z 0 [System] [MY-010910] [Server] /opt/bitnami/mysql/bin/mysqld.bin: Shutdown complete (mysqld 8.0.18)  MySQL Community Server - GPL.
2020-11-16T04:36:57.207428Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T04:36:57.214516Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 1903
2020-11-16T04:38:00.901675Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T04:38:00.904008Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 2417
2020-11-16T04:40:43.575649Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T04:40:43.578652Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 3721
2020-11-16T05:22:45.422049Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T05:22:45.426917Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 1915
2020-11-16T05:25:07.601525Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T05:25:07.633171Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 2632
2020-11-16T05:27:06.971748Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T05:27:06.975053Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 3064
2020-11-16T05:28:13.972913Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T05:28:13.976067Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 3482
terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable
05:28:14 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x5ad46a0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fc46fbead00 thread_stack 0x46000
2020-11-16T05:31:06.593769Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T05:31:06.596681Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 4007
2020-11-16T05:31:07.400988Z 1 [Warning] [MY-012637] [InnoDB] 1048576 bytes should have been written. Only 520192 bytes written. Retrying for the remaining bytes.
2020-11-16T05:31:07.402204Z 1 [Warning] [MY-012638] [InnoDB] Retry attempts for writing partial data failed.
2020-11-16T05:31:07.402798Z 1 [ERROR] [MY-012639] [InnoDB] Write to file ./ibtmp1 failed at offset 8388608, 1048576 bytes should have been written, only 520192 were written. Operati2020-11-16T05:31:07.854043Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2020-11-16T05:31:07.855923Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2020-11-16T05:31:07.856259Z 0 [ERROR] [MY-010119] [Server] Aborting
2020-11-16T05:31:07.876809Z 0 [System] [MY-010910] [Server] /opt/bitnami/mysql/bin/mysqld.bin: Shutdown complete (mysqld 8.0.18)  MySQL Community Server - GPL.
2020-11-16T06:19:55.215539Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T06:19:55.244923Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 2033
2020-11-16T06:20:03.861135Z 0 [ERROR] [MY-012681] [InnoDB] mmap(17170432 bytes) failed; errno 12
2020-11-16T06:20:04.094948Z 1 [ERROR] [MY-012956] [InnoDB] Cannot allocate memory for the buffer pool
2020-11-16T06:20:04.583159Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2020-11-16T06:20:05.036583Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2020-11-16T06:20:05.524939Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2020-11-16T06:20:05.546593Z 0 [ERROR] [MY-010119] [Server] Aborting
2020-11-16T06:20:06.259020Z 0 [System] [MY-010910] [Server] /opt/bitnami/mysql/bin/mysqld.bin: Shutdown complete (mysqld 8.0.18)  MySQL Community Server - GPL.
2020-11-16T06:20:10.483019Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T06:20:10.485595Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 2416
2020-11-16T06:23:00.497787Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T06:23:00.498502Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 3462
2020-11-16T06:37:51.464841Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T06:37:51.471094Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 1929
2020-11-16T06:42:12.070643Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T06:42:12.075883Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 3397
2020-11-16T06:44:12.017208Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T06:44:12.020089Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 3827
2020-11-16T06:46:12.085649Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T06:46:12.090319Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 4248
2020-11-16T06:48:12.071822Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T06:48:12.077780Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 4700
2020-11-16T06:51:43.288138Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-11-16T06:51:43.294069Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 5191

Please help fixing this issue

Hi,

I also ran the Bitnami SupportTool but, I got the following message:
The support bundle file was successfully created, but the automatic upload to Bitnami servers failed. You will need to upload it to your Bitnami Support ticket manually. Please locate the following file in your file browser or in your terminal: /tmp/bitnami-wordpress-info-20201116-085706-11764.zip Exercise caution when uploading the resulting support bundle to public sites as it contains detailed diagnostic information.

Can I upload the zip file here?

Hi @pramgopal2002,

It seems there is a performance problem in the instance. I just sent you a private message with the instructions to share the .zip file.

Apart from that, can you run these commands and share the output here?

df -H
free -m

Thanks

root@ip-172-26-2-191:/home/bitnami# sudo df -h
Filesystem Size Used Avail Use% Mounted on
udev 236M 0 236M 0% /dev
tmpfs 49M 5.7M 43M 12% /run
/dev/xvda1 20G 19G 1.1G 95% /
tmpfs 244M 0 244M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 244M 0 244M 0% /sys/fs/cgroup
/dev/loop0 98M 98M 0 100% /snap/core/10126
/dev/loop1 29M 29M 0 100% /snap/amazon-ssm-agent/2333
/dev/loop2 98M 98M 0 100% /snap/core/10185
/dev/loop3 56M 56M 0 100% /snap/core18/1932
/dev/loop4 29M 29M 0 100% /snap/amazon-ssm-agent/2012
/dev/xvdf 32G 48M 30G 1% /data
tmpfs 49M 0 49M 0% /run/user/1000

root@ip-172-26-2-191:/home/bitnami# free -m
total used free shared buff/cache available
Mem: 486 113 242 63 130 267
Swap: 0 0 0

Also, I shared the Bitnami Support Tool file

Hi,

Any update on this issue. Please help.

Thanks

Hi @pramgopal2002,

You are using a really small instance in Lightsail and its performance is really poor. We suggest you increase the instance type to the one with 1GB to solve this issue. If not, you will continue running into this problem in the future.

Hi Jota,

Thanks for the reply.
Upgraded to the one with 2GB RAM and still mysql won’t start.

bitnami@ip-172-26-14-95:~$ sudo /opt/bitnami/ctlscript.sh restart
Syntax OK
/opt/bitnami/apache2/scripts/ctl.sh : apache not running
/opt/bitnami/php/scripts/ctl.sh : php-fpm stopped
/opt/bitnami/mysql/scripts/ctl.sh : mysql not running
/opt/bitnami/mysql/scripts/ctl.sh : mysql  could not be started
/opt/bitnami/php/scripts/ctl.sh : php-fpm started
Syntax OK
/opt/bitnami/apache2/scripts/ctl.sh : httpd started at port 80

I created the new instance from the snapshot that was created after the MYSQL issue came up, as I don’t have any old snapshot.

Please help resolving this MYSQL issue.
Thanks

Please run the Bitnami Support tool again so we get updated information from your system to investigate the database’s problems.

Please note that you need to paste the code ID that is shown at the end.

Hi Jota,

Thanks for the reply.
I was unable to get the code ID as it was unable to upload to the server. So I sent you the file in the private thread. Please check and let me know any instructions to fix this issue.

Regards

Hi @pramgopal2002,

I didn’t receive any private message with the link and I can see you didn’t send any private message either. Please make sure you use the private thread I started some days ago to share this file.

Thanks

Hi Jota,

Sorry, The message was not sent earlier and I just posted it with the link to the file. Please check.

Thanks

Hi @pramgopal2002,

If you take a look at the MySQL’s log file, you can find this error

2020-11-19T07:57:13.750664Z 0 [ERROR] [MY-000067] [Server] unknown variable 'sql_log_bin=0'.
2020-11-19T07:57:13.751075Z 0 [ERROR] [MY-010119] [Server] Aborting

I saw that you modified the MySQL’s configuration

sql_log_bin = 0

and that’s why the database is failing. Please remove that line and follow this guide if you want to remove the binary logging in the database

https://docs.bitnami.com/aws/apps/wordpress/troubleshooting/disable-binary-logging-mysql/

Hi Jota,

That solved the issue and MySQL is running now.

Thanks a lot.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.