Support Bundle Uploaded ID: 3c41e560-2865-1432-0a99-02687bf13930

aqeel.hossain 2020-09-10 02:51:05 UTC #1

Keywords: Nginx - Virtual Machines - Technical issue - Other
bnsupport ID: 3c41e560-2865-1432-0a99-02687bf13930
Description:
The site is down due to "Error establishing a database connection" I checked MySql is down and unable to start or restart also sda1 drive is almost full below is the device space status:

root@bitnami-wordpresspro-4300:/home/aqeel116/stack# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 210K 278 210K 1% /dev
tmpfs 213K 557 213K 1% /run
/dev/sda1 640K 96K 545K 15% /
tmpfs 213K 1 213K 1% /dev/shm
tmpfs 213K 9 213K 1% /run/lock
tmpfs 213K 15 213K 1% /sys/fs/cgroup
root@bitnami-wordpresspro-4300:/home/aqeel116/stack# df -h
Filesystem Size Used Avail Use% Mounted on
udev 840M 0 840M 0% /dev
tmpfs 171M 8.1M 162M 5% /run
/dev/sda1 9.8G 9.3G 1.6M 100% /
tmpfs 851M 0 851M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 851M 0 851M 0% /sys/fs/cgroup

also, I am trying a password I know in SSH Console but it's not accepted

aqeel.hossain 2020-09-10 04:39:34 UTC #2

Now I managed to delete unwanted files and reduced the disk space in use current status is below yet mysql is not starting i also tried rebooting the vm. still site in down with error "Error establishing a database connection"

aqeel116@bitnami-wordpresspro-4300:~$ sudo /opt/bitnami/ctlscript.sh status
Nginx already running
php-fpm already running
mysql not running
aqeel116@bitnami-wordpresspro-4300:~$ sudo /opt/bitnami/ctlscript.sh start
/opt/bitnami/mysql/scripts/ctl.sh : mysql could not be started
/opt/bitnami/php/scripts/ctl.sh : php-fpm (pid 1435) already running
/opt/bitnami/nginx/scripts/ctl.sh : Nginx (pid 1448) already running
aqeel116@bitnami-wordpresspro-4300:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 840M 0 840M 0% /dev
tmpfs 171M 3.9M 167M 3% /run
/dev/sda1 9.8G 6.2G 3.1G 67% /
tmpfs 851M 0 851M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 851M 0 851M 0% /sys/fs/cgroup

davidg 2020-09-10 10:56:58 UTC #3

Hello @aqeel.hossain,

As you said, the first error we can check in your mysqld.log file (from the first message) is:

2020-09-10T02:36:08.280264Z 1 [ERROR] [MY-012640] [InnoDB] Error number 28 means 'No space left on device'
2020-09-10T02:36:08.280499Z 1 [ERROR] [MY-012267] [InnoDB] Could not set the file size of './ibtmp1'. Probably out of disk space
2020-09-10T02:36:08.280626Z 1 [ERROR] [MY-012926] [InnoDB] Unable to create the shared innodb_temporary.
2020-09-10T02:36:08.280755Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2020-09-10T02:36:08.515249Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2020-09-10T02:36:08.515527Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2020-09-10T02:36:08.515796Z 0 [ERROR] [MY-010119] [Server] Aborting

Could you check if that file shows a different error or run the support tool again so we can check it?

Regards

aqeel.hossain 2020-09-10 13:49:13 UTC #4

Dear @davidg,

Hope you are doing great here is the latest support code 64e0c809-7693-cfac-ede1-a45502fa1695

Please review so that my site can become up.

Regards
Aqeel

aqeel.hossain 2020-09-10 14:44:30 UTC #5

Also i have tried to start MySql manually and here is the result that i found below:

sudo $MYSQL_START
2020-09-10T14:32:39.859285Z mysqld_safe Logging to '/opt/bitnami/mysql/data/mysqld.log'.
2020-09-10T14:32:39.877280Z mysqld_safe Starting mysqld.bin daemon with databases from /opt/bitnami/mysql/data
2020-09-10T14:32:41.840026Z mysqld_safe mysqld from pid file /opt/bitnami/mysql/data/mysqld.pid ended

aqeel116@bitnami-wordpresspro-4300:~$ sudo su
root@bitnami-wordpresspro-4300:/home/aqeel116# sudo /opt/bitnami/ctlscript.sh status
Nginx already running
php-fpm already running
mysql not running
root@bitnami-wordpresspro-4300:/home/aqeel116# sudo /opt/bitnami/ctlscript.sh start
/opt/bitnami/mysql/scripts/ctl.sh : mysql could not be started
/opt/bitnami/php/scripts/ctl.sh : php-fpm (pid 14403) already running
/opt/bitnami/nginx/scripts/ctl.sh : Nginx (pid 14415) already running
root@bitnami-wordpresspro-4300:/home/aqeel116#

davidg 2020-09-11 08:50:16 UTC #6

Hi @aqeel.hossain,

I can see this:

mysqld.bin: File './binlog.0000X' not found (OS errno 2 - No such file or directory)
2020-09-10T04:23:45.764049Z 0 [ERROR] [MY-010958] [Server] Could not open log file.
2020-09-10T04:23:45.764222Z 0 [ERROR] [MY-010041] [Server] Can't init tc log
2020-09-10T04:23:45.764464Z 0 [ERROR] [MY-010119] [Server] Aborting

Could you take a look at this previous case?

Regards

aqeel.hossain 2020-09-11 17:05:59 UTC #7

Hi davidg

i tried but its not working for me please review why mysql isnt starting

regards

aqeel.hossain 2020-09-12 21:47:00 UTC #8

Hi @davidg
I am stuck with it and need your serious support mysite is down now from last 7+ days and i am unable to do anything despite continuous followup and self-research and try . Please review the case and let me know asap . I have also tried to check on google cloud platform dashboard but didnt understood anything.

aqeel.hossain 2020-09-13 07:20:21 UTC #9

Dear @davidg,

here is my.cnf file content below:

[mysqladmin]
user=root

[mysqld]
basedir=/opt/bitnami/mysql
datadir="/opt/bitnami/mysql/data"
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
tmpdir=/opt/bitnami/mysql/tmp
max_allowed_packet=32M
bind-address=127.0.0.1
skip-name-resolve=1
expire_logs_days=7
character-set-server=utf8mb4
collation-server=utf8mb4_general_ci

default-authentication-plugin=mysql_native_password
[client]
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock

default-character-set=utf8mb4
[manager]
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
pid-file=/opt/bitnami/mysql/tmp/manager.pid
default-mysqld-path=/opt/bitnami/mysql/bin/mysqld.bin

!include /opt/bitnami/mysql/bitnami/my.cnf

there is no entry in my.cnf that says sql_log_bin or set_log_bin . I tried add the same entry on my own into the file yet not of any help.

tried 2nd command and here is the return:
aqeel116@bitnami-wordpresspro-4300:~$ mysql -u root -p
Enter password:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/bitnami/mysql/tmp/mysql.sock' (2)
aqeel116@bitnami-wordpresspro-4300:~$

also on root it returned same results. I tried making new sql instance in google cloud platform and installed mysql yet nothing below i tried stopping all services and then restarting them but mysql again failed to start.
root@bitnami-wordpresspro-4300:/opt/bitnami/mysql/bitnami# sudo /opt/bitnami/ctlscript.sh stop
/opt/bitnami/nginx/scripts/ctl.sh : Nginx stopped
/opt/bitnami/php/scripts/ctl.sh : php-fpm stopped
/opt/bitnami/mysql/scripts/ctl.sh : mysql not running
root@bitnami-wordpresspro-4300:/opt/bitnami/mysql/bitnami# sudo /opt/bitnami/ctlscript.sh status
Nginx not running
php-fpm not running
mysql not running
root@bitnami-wordpresspro-4300:/opt/bitnami/mysql/bitnami# sudo /opt/bitnami/ctlscript.sh restart
/opt/bitnami/nginx/scripts/ctl.sh : Nginx not running
/opt/bitnami/php/scripts/ctl.sh : php-fpm not running
/opt/bitnami/mysql/scripts/ctl.sh : mysql not running
/opt/bitnami/mysql/scripts/ctl.sh : mysql could not be started
/opt/bitnami/php/scripts/ctl.sh : php-fpm started
/opt/bitnami/nginx/scripts/ctl.sh : Nginx started
root@bitnami-wordpresspro-4300:/opt/bitnami/mysql/bitnami#

Please do the need full and confirm.
Here is a new support ID I just collected again. 11b854d7-b110-1288-33a8-4b3d8dd80fee

marcos 2020-09-14 13:12:03 UTC #10

Hi @aqeel.hossain, the MySQL logs indeed seem to indicate your disk got full:

2020-09-09T15:48:08.191589Z 1 [ERROR] [MY-012639] [InnoDB] Write to file ./ibtmp1 failed at offset 11534336, 1048576 bytes should have been written, only 585728 were written. Operating system error number 28. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.
2020-09-09T15:48:08.191714Z 1 [ERROR] [MY-012640] [InnoDB] Error number 28 means 'No space left on device'
2020-09-09T15:48:08.191949Z 1 [ERROR] [MY-012267] [InnoDB] Could not set the file size of './ibtmp1'. Probably out of disk space

After that error, the following started showing:

2020-09-10T04:09:28.203615Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 10844
mysqld.bin: File './binlog.000017' not found (OS errno 2 - No such file or directory)
2020-09-10T04:09:28.959944Z 0 [ERROR] [MY-010958] [Server] Could not open log file.
2020-09-10T04:09:28.960107Z 0 [ERROR] [MY-010041] [Server] Can't init tc log
2020-09-10T04:09:28.960376Z 0 [ERROR] [MY-010119] [Server] Aborting

So it does look like your DB got corrupted as the "binlog.000017" file is not present.

To fix that, please take a look at the previous thread mentioned by @davidg https://community.bitnami.com/t/mysql-is-not-running/70417/10 and follow these steps:

Edit the file: /opt/bitnami/mysql/data/binlog.index
Remove the line: ./binlog.000017
Start MySQL again. New entries in the MySQL log file (/opt/bitnami/mysql/data/mysqld.log) should not show the error "File './binlog.000017' not found" anymore.
If you see any error like "File './binlog.0000xx' not found", go to the first step and repeat.

NOTE: Please make a backup of the entire Bitnami installation before modifying the MySQL index file.

aqeel.hossain 2020-09-16 03:12:04 UTC #11

Thanks @davidg & @marcos ..at last managed to get my site live again with your support. Luv u both Bro

aqeel.hossain 2020-09-19 23:35:43 UTC #12

Dear @davidg & @marcos

Still need your support in it without doing anything my site is filling its disk space on its own around 1 GB in 24 hours . I am afraid site will be down soon again..Do i have to keep fire fighting the disk space all the time to keep the site up ? Or do we have any permanent solution to this auto disk filling most probably via sql Logs.

aqeel116@bitnami-wordpresspro-4300:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 840M 0 840M 0% /dev
tmpfs 171M 19M 152M 12% /run
/dev/sda1 9.8G 8.4G 900M 91% /
tmpfs 851M 0 851M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 851M 0 851M 0% /sys/fs/cgroup

Here is the current support code to help you understand the issue: cec16985-208c-455c-daf2-d68e8d427a20

regards,

aqeel.hossain 2020-09-20 07:33:06 UTC #13

Moreover Brothers, I am unable to remove the Bitnami Banner following is error need your support:

Bitnami WordPress with NGINX and SSL Stack Frequently Run Commands > Remove the Bitnami Banner

Something when wrong while removing the banner, Please run bnsupport and open a ticket at: https://community.bitnami.com Error ocurred:exit status 1

davidg 2020-09-21 07:14:08 UTC #14

Hello @aqeel.hossain,

Since this is another topic, could you please open another case with this information? The idea is to avoid mixing topics in the forum.

Regards

system 2020-10-05 07:14:08 UTC #15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!