Mysqld.bin CPU usage between 150% - 200% with Bitnami + Wordpress + LAMP on AWS EC2

Keywords: WordPress - AWS - Technical issue - Other
Description:
Hi Guys,

Need help understanding what the issue is with the “mysqld.bin” process constantly running at 150% to 200% CPU usage.

The site had no issues until about a week or so, before was running fine.

I have a LAMP Bitnami configuration on AWS EC2
I’ve already gone thru similar threads here and tried a few things but to no avail.
I’ve also followed the troubleshooting guides offered by bitnatmi docs.
I’m pasting the outputs here.
Please guide me to fix this issue.

# ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS

Output:

  112 /sbin/iscsid
  164 /sbin/mdadm --monitor --pid-file /run/mdadm/monitor.pid --daemonise --scan --syslog
  680 pr -TW167
  720 sort -b -k1,1n
 1220 /sbin/lvmetad -f
 1312 /usr/sbin/acpid
 1476 ps -e -orss=,args=
 1704 /bin/sh /opt/bitnami/mysql/bin/mysqld_safe --defaults-file=/opt/bitnami/mysql/my.cnf --mysqld=mysqld.bin --socket=/opt/bitnami/mysql/tmp/mysql.sock --datadir=/op
 1960 /usr/sbin/atd -f
 2000 (sd-pam)
 2372 /lib/systemd/systemd-timesyncd
 2708 /usr/bin/lxcfs /var/lib/lxcfs/
 2752 /usr/sbin/cron -f
 2784 /sbin/dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0
 2840 /usr/sbin/rsyslogd -n
 2892 /lib/systemd/systemd-logind
 3048 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
 3132 /lib/systemd/systemd --user
 3212 /usr/sbin/sshd -D
 3244 /lib/systemd/systemd-udevd
 3356 /lib/systemd/systemd-journald
 3508 /sbin/iscsid
 4084 /usr/lib/policykit-1/polkitd --no-debug
 4320 /usr/bin/gonit
 4552 /usr/lib/accountsservice/accounts-daemon
 4596 /sbin/init
 4744 /usr/sbin/haveged --Foreground --verbose=1 -w 1024
 4752 sshd: ubuntu@pts/0
 4808 sshd: ubuntu [priv]
 5328 -bash
 6148 /snap/amazon-ssm-agent/1480/amazon-ssm-agent
 9640 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf)
13700 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
14156 /usr/lib/snapd/snapd
37632 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
46708 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
52788 php-fpm: pool wordpress
52804 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
52992 php-fpm: pool wordpress
54168 php-fpm: pool wordpress
57380 php-fpm: pool wordpress
58508 php-fpm: pool wordpress
59172 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
62540 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
438516 /opt/bitnami/mysql/bin/mysqld.bin --defaults-file=/opt/bitnami/mysql/my.cnf --basedir=/opt/bitnami/mysql --datadir=/opt/bitnami/mysql/data --plugin-dir=/opt/bit

# ps -e -o pcpu,nice,state,cputime,args --sort -pcpu | head -10

Output:-

%CPU  NI S     TIME COMMAND
 99.0   0 S 00:01:04 /opt/bitnami/mysql/bin/mysqld.bin --defaults-file=/opt/bitnami/mysql/my.cnf --basedir=/opt/bitnami/mysql --datadir=/opt/bitnami/mysql/data --plugin-dir=/opt/bitnami/mysql/lib/plugin --user=mysql --lower-case-table-names=1 --log-error=/opt/bitnami/mysql/data/mysqld.log --pid-file=/opt/bitnami/mysql/data/mysqld.pid --socket=/opt/bitnami/mysql/tmp/mysql.sock --port=3306
 0.1   0 S 00:00:04 php-fpm: pool wordpress
 0.1   0 S 00:00:03 php-fpm: pool wordpress
 0.1   0 S 00:00:03 php-fpm: pool wordpress
 0.1   0 S 00:00:03 php-fpm: pool wordpress
 0.1   0 S 00:00:00 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
 0.0   0 S 00:00:08 /sbin/init
 0.0   0 S 00:00:00 [kthreadd]
 0.0   0 S 00:00:05 [ksoftirqd/0]


# df -ih

Output:

Filesystem     Inodes IUsed IFree IUse% Mounted on
udev             248K   322  248K    1% /dev
tmpfs            250K   413  250K    1% /run
/dev/xvda1       5.5M  251K  5.3M    5% /
tmpfs            250K     1  250K    1% /dev/shm
tmpfs            250K     3  250K    1% /run/lock
tmpfs            250K    16  250K    1% /sys/fs/cgroup
/dev/loop1         15    15     0  100% /snap/amazon-ssm-agent/1480
/dev/loop0         15    15     0  100% /snap/amazon-ssm-agent/1455
/dev/loop2        13K   13K     0  100% /snap/core/7917
/dev/loop3        13K   13K     0  100% /snap/core/7713
tmpfs            250K     4  250K    1% /run/user/1000

Appreciate all the help I can get.

Thank you.

Hi @viveks,

We have a Support Tool that will gather relevant information for us to analyze your configuration and logs. Could you please download and execute it on the machine where the stack is running by following the steps described in the guide below?

How to Run the Bitnami Support Tool

Please note that you need to paste the code ID that is shown at the end.

Regards,
Michiel

Hi Michiel,

Thank you for the quick response.

Please find the code ID: 83d58d7a-d7e0-7e8a-906a-5297f262fe30

Hi @viveks,

I just checked the log file of MySQL (mysql/data/mysqld.log) and found that there were too many restarts of the MySQL server. Can you check the log file and see if those restarts were performed when you noticed that the CPU usage was high?

Apart from that, can you check if a bot is “attacking” your site? Maybe those requests are making MySQL crash.

https://docs.bitnami.com/aws/apps/wordpress/troubleshooting/deny-connections-bots/

Thanks

Hi Jota,

Yes you are right. mysql restart which makes the log looks like.

Examined the bot attack as per the link shared, but nothing found.

When i click on the website it works fine.when i click 4th tab (any link on the website ) it starts slowingup, the same time server load starts to increase.

result:
5091 mysql 20 0 1500560 431572 28696 S 80.0 21.1 2:25.04 /opt/bitnami/mysql/bin/mysqld.bin --defaults-file=/opt/bitnami/mysql/my.cnf --basedir=/opt/bitnami

ok @viveks,

I’ll forward this ticket to our engineering team so they can take a look at all the information. We will update this topic once we have more information

@jota Thank you. Waiting for your valuable response.

Bitnami Engineers

Any updates please.

Hi @viveks

I noticed you have several plugins installed (check the list below), are you using all of them? Did you install or activate any of them recently?

accesspress-social-share
akismet
bis-mega-enterprise-platform
bis-redirect-controller-addon
force-regenerate-thumbnails
password-protected
post-views-counter
profile-builder
reading-time-wp
responsive-lightbox
restrict-login-by-ip
slider-by-supsystic
sucuri-scanner
two-factor-authentication
wordpress-seo
wp-ulike
yet-another-related-posts-plugin

You can try to enable the slow queries log (I explained how to do it on this thread) so you can inspect/identify why MySQL is so slow. In the link below you can find some guidance to debug issues by inspecting slow queries:

https://serverfault.com/questions/418/how-do-you-track-and-debug-mysql-performance-issues

Another thing you can try to reduce the MySQL usage is to use Memcached (which is not installed by default in the WP stack) and some plugin (such as wp-total-cache) to reduce the number of requests that are handled by MySQL server.

Best regards,

Juan Ariza

I have a similar problem on my Amazon Lightsail instance even though I am not using Wordpress, I am still stuck in finding a solution but I feel the problem is that Amazon is overselling the physical servers. Can you check with SAR command the %of steal CPU? In my case here the server get stuck when the %steal is running high.

Link to SAR tool

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.