Moodle Stack - Mysql Crashes

Keywords: Moodle - Linux - Technical issue - Other
Description:
Hi

My bitnami support-tool package is uploaded and ready to share as it has failed to generate automated one.

I am using bitnami moodle stack version 3.8.1
My specs are:
Vhost with : 8 vcpu - 32g Ram - 160G SSD

Recently, MySql module crashes randomly and I couldn’t stop it via ctlscript.sh. The only temporary fix I can do is to pkill sql and then start the module with ctlscript.sh start mysql

This has been happening quite randomly between once a week or even twice in a minute.

Some part of the log file located in /opt/bitnami/mysql/data/mysql.lost :

--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: complete io for buf page (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
 ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 1; buffer pool: 9
1277615 OS file reads, 8971122 OS file writes, 3688491 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 2202, seg size 2204, 109905 merges
merged operations:
 insert 107441, delete mark 98897, delete 18341
discarded operations:
 insert 0, delete mark 0, delete 0
Hash table size 69257, node heap has 40 buffer(s)
Hash table size 69257, node heap has 125 buffer(s)
Hash table size 69257, node heap has 126 buffer(s)
Hash table size 69257, node heap has 34 buffer(s)
Hash table size 69257, node heap has 105 buffer(s)
Hash table size 69257, node heap has 112 buffer(s)
Hash table size 69257, node heap has 65 buffer(s)
Hash table size 69257, node heap has 49 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number          33964264658
Log buffer assigned up to    33964264658
Log buffer completed up to   33964264658
Log written up to            33964264658
Log flushed up to            33964239614
Added dirty pages up to      33964264658
Pages flushed up to          33964239604
Last checkpoint at           33964232016
4353846 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 274726912
Dictionary memory allocated 3457656
Buffer pool size   16384
Free buffers       1022
Database pages     14682
Old database pages 5399
Modified db pages  61
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 4065533, not young 30310518
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1276947, created 44976, written 4346897
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 14682, unzip_LRU len: 36
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=18651, Main thread ID=139684704597760 , state=enforcing dict cache limit
Number of rows inserted 1037592, updated 749554, deleted 299860, read 4792357429
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
InnoDB: ###### Diagnostic info printed to the standard error stream
2020-12-02T07:20:30.755837Z 0 [ERROR] [MY-012872] [InnoDB] [FATAL] Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
2020-12-02T07:20:30.755925Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: ut0ut.cc:532 thread 139684736861952
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
07:20:30 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x46000
/opt/bitnami/mysql/bin/mysqld.bin(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x1ed6cce]
/opt/bitnami/mysql/bin/mysqld.bin(handle_fatal_signal+0x323) [0xfb2d23]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0) [0x7f0b1c66f0e0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf) [0x7f0b1a6dafff]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f0b1a6dc42a]
/opt/bitnami/mysql/bin/mysqld.bin() [0xd2ef58]
/opt/bitnami/mysql/bin/mysqld.bin(ib::fatal::~fatal()+0x95) [0x2171ca5]
/opt/bitnami/mysql/bin/mysqld.bin(srv_error_monitor_thread()+0x81f) [0x2115d0f]
/opt/bitnami/mysql/bin/mysqld.bin(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Runnable, void (*)()> > >::_M_run()+0xa5) [0x202eac5]
/opt/bitnami/mysql/bin/mysqld.bin() [0x25d3f0f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4) [0x7f0b1c6654a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f0b1a790d0f]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2020-12-02T07:23:14.564211Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2020-12-02T07:23:14.564454Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 14645
2020-12-02T07:23:19.566947Z 0 [System] [MY-010229] [Server] Starting crash recovery...
2020-12-02T07:23:20.853099Z 0 [System] [MY-010232] [Server] Crash recovery finished.
2020-12-02T07:23:22.088294Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2020-12-02T07:23:22.148346Z 0 [System] [MY-010931] [Server] /opt/bitnami/mysql/bin/mysqld.bin: ready for connections. Version: '8.0.18'  socket: '/opt/bitnami/mysql/tmp/mysql.sock'  port: 3306  MySQL Community Server - GPL.
2020-12-02T07:23:22.170188Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/tmp/mysqlx.sock' bind-address: '::' port: 33060

Correction : Some part of the log file located in /opt/bitnami/mysql/data/mysql.log

Hi @mobcer,

Can you run the bnsupport tool and send us the code ID it outputs in the end?

sudo /opt/bitnami/bnsupport-tool

Regards,
Michiel

bnsupport tool output been sent in direct message

Regards,

Hi @mobcer,

The bnsupport information you shared is from September and I understand the error you are getting is not the same one you got that month. In that case, you ran into a “No space left on device” error. That means that you didn’t have enough free space in the machine. What about now? Do you have free space?

df -H

You can run the Bitnami Support tool again and ensure you get the latest file or look for the file bnsupport generated during the last execution (it should look like bitnami-moodle-info-20201208-...) and share it.

Thanks

Hi @jota . Good to see you again :slight_smile:
The link you have probably is the last one that you helped me with within September :+1: . The information I’ve shared with @michiel was newer that I also shared with you now.

Hi @mobcer,

The file is the one from September. Please notice the date in the name. Are you sure you are sharing the correct URL?

Hi

Sorry it was my bad. got confused by multiple files on drive. I’ve sent the new files

Regards

I just took a look at your files and found many warnings like this one in the MySQL’s log file

2020-12-06T05:25:27.910315Z 0 [Warning] [MY-010909] [Server] /opt/bitnami/mysql/bin/mysqld.bin: Forcing close of thread 6312  user: 'bn_moodle'.

it seems to be a problem with the application itself

https://moodle.org/mod/forum/discuss.php?d=75030

If that doesn’t work for you, you can ask the Moodle’s support team to know more about the problem.

Thanks

Thank you @jota for pointing that out.

I checked all tables in bitnami_moodle.mdl database with mysqlcheck. Although I also was suspicious of table corruption, nonetheless found no errors.

Hi @mobcer,

Did you ask the Moodle’s developers to try to get more information of the error I saw in the MySQL’s log file?

1 Like

Hi @jota

I edited my.cnf mysql file and increased max_connections value to 1500 connections. So far has not been crashed.

Hi @mobcer,

thanks for sharing your solution.
As it seems solved I will close the topic, do not hesitate to write us back if you have any other questions.

Regards,
Ibone.