HD space is being filled

Keywords: WordPress - AWS - Technical issue - Other

bnsupport ID: 475ff90a-8f16-319a-2bb9-28dadb0c8712

bndiagnostic output:

? Apache: Found possible issues
? Mysql: Found possible issues
? Php: Found possible issues
https://docs.bitnami.com/general/apps/wordpress/troubleshooting/debug-errors-apache/
https://docs.bitnami.com/bch/apps/moodle/troubleshooting/deny-connections-bots-apache/
https://docs.bitnami.com/general/apps/wordpress/troubleshooting/disable-binary-logging-mysql/
https://docs.bitnami.com/general/apps/wordpress/configuration/configure-phpfpm-processes/

bndiagnostic failure reason: The documentation did not make any significant change

Description:
/dev/xvda1 630G 266G 339G 45% /

this server was back up snap shot from aws lightsail, but i had to use a snap shot because my old server shut down because it was being filled to 100%

Hi @rpak,

Thanks for using Bitnami. The bndiagnostic tool found some errors, did you have any chance to check the suggested links? According to the retrieved data, you are getting a big number of requests from a few IP addresses. About this, we recommend you to check the guide about bots attack in order to detect and block if any IP address is doing an attack against your website

https://docs.bitnami.com/bch/apps/moodle/troubleshooting/deny-connections-bots-apache/

These are the numbers the tool retrieved regarding this topic. Each line shows the number of requests performed from a unique IP address

-----------------------------------
Check performance issues: Count number of requests for the 10 most active IP addresses in the last 100.000 requests
-----------------------------------
Running: tail -n 100000 access_log | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10 | awk '{print $1}'
In: /opt/bitnami/apache2/logs/

Output:

13029
12834
12737
2971
2361
889
883
678
646
642

Apart from that, I see you have many plugins installed on your website, and we found in the past that it generates issues when the plugins use the database to store temporal data. All those database access create events that get logged in the binlog files, making them increase fast. According to the bnsupport data, this seems to be your case as well. Please check our docs linked below on how to disable the binary logging

https://docs.bitnami.com/aws/infrastructure/mysql/troubleshooting/disable-binary-logging-mysql/

Additionally, you can run a MySQL command to force the purge of the current binary logs

https://dev.mysql.com/doc/refman/8.0/en/purge-binary-logs.html

Hi Gongomgra,

Thank you for this reponse I have done all you have request of me for the binlogs but they still appear to be in my harddrive. The SQL purge did not work.

https://gyazo.com/29000a22e1c9e2c6686f32d7b5c11dee

they are still all in the /opt/bitnami/mysql/data/bin

there are over 300 binlog files still in there.

I used both purge queries

PURGE BINARY LOGS TO ‘mysql-bin.010’;
PURGE BINARY LOGS BEFORE ‘2019-04-02 22:46:26’;

I have changed the date and bin number according to my case but nothing was removed

Also is there a way to figure out what is causing the issue? Like which plug in is creating all these binlogs or is the only way to find out to disable and run them one by one.

Please see following image to see the binlogs still in folder data

Hi @rpak,

Thanks for your message.

Also is there a way to figure out what is causing the issue? Like which plug in is creating all these binlogs or is the only way to find out to disable and run them one by one.

There is a tool provided by the MySQL team you can use to investigate all those database events in order to find the plugin causing the big files, but you need to use it before removing the binlog information.

https://dev.mysql.com/doc/refman/8.0/en/mysqlbinlog.html

Also notice the commands shown in the MySQL docs are just a couple of examples to show you how to run a sample command, but you need to update the values to reflect your current situation. For instance, your files are named binlog.xyz, not mysql-bin.xyz, so maybe using another pattern works.

Before doing any other change on your server, we highly recommend you to create a server backup that you can easily restore in case of need.

https://docs.bitnami.com/aws/faq/administration/backup-restore-server/

Once you have created your server backup, can you try the next command in the MySQL console? It should purge all the binary files created before September the 6th at 10:00 AM

PURGE BINARY LOGS BEFORE '2021-09-06 10:00:00';

so i am writing mysqlbinlog binlog.000844

root@ip-172-26-14-106:/opt/bitnami/mysql/data# mysqlbinlog binlog.000844
mysqlbinlog.bin: [ERROR] unknown variable 'default-character-set=utf8mb4'.
root@ip-172-26-14-106:/opt/bitnami/mysql/data# mysqlbinlog 
mysqlbinlog.bin: [ERROR] unknown variable 'default-character-set=utf8mb4'.
root@ip-172-26-14-106:/opt/bitnami/mysql/data# mysqlbinlog log
mysqlbinlog.bin: [ERROR] unknown variable 'default-character-set=utf8mb4'.
root@ip-172-26-14-106:/opt/bitnami/mysql/data# mysqlbinlog binlog.000844
mysqlbinlog.bin: [ERROR] unknown variable 'default-character-set=utf8mb4'.
root@ip-172-26-14-106:/opt/bitnami/mysql/data# 

and thats what im getting

Hello @rpak,

It seems mysqlbinlog is having an issue using the character configuration obtained from my.cnf. Please try executing the logging tool using the --no-defaults option:

mysqlbinlog --no-defaults binlog.000844

Regards,
Francisco de Paz

Ty @fdepaz

So this is what I am getting when I try to read the binlog, my hardrive even with the expiration settings seem to still be growing. I disabled binlog but that also caused errors on my server so I had to spin up a new snap shot without it disabled.

Is there a way I can find out which plug in is generating these binglogs or where the binlogs are coming from.

Hello @rpak,

Unfortunately, there is no easy option to pinpoint the faulty plugin. To do so, I would recommend to start disabling your plugins until you find the culprit. Apart from that, if the option disable_log_bin is failing, could you please try with skip-log-bin? A stop-gap would be to limit the time the binlogs are saved. You can also use an option in my.cnf to configure this:

[mysqld]
skip-log-bin
binlog_expire_logs_seconds= 345600

After that, restart your database and check whether the changes were applied.

Regards,
Francisco de Paz

Hello I am contacting to see why my DSK xvda is busy at 100% and how to fix this, I am using the atop program as told before

How do I fix this