(!) CPU Burst Overload

Keywords: MySQL - AWS - Technical issue - Other
bnsupport ID: a7c6d9e8-3181-1369-9076-fa85bf1a03e4
Description:
Hello,

our website is going down because of CPU burst. Can you explain to me, how to figure out what process is attacking us down? I already tried some things like disable OpCache without success.

Situation: Wordpress Instance
Server: Lightsail (highest AWS Stack Server avalaible 4 CPU + 16 GB RAM)

php-fpm: and mysqld.bin is heavy loading!

Hi @eSpox,

We have a guide here that explains how to investigate if there is any bot/attacker accessing your site with malicious intentions

https://docs.bitnami.com/aws/apps/wordpress/troubleshooting/deny-connections-bots-apache/

If you do not see anything weird there, I also saw that PHP-FPM is reaching the limits, can you take a look at the PHP-FPM’s log file (/opt/bitnami/php/var/log/php-fpm.log)?

[08-Jun-2021 11:06:17] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 5 idle, and 26 total children

You can take a look at these guides to update those values

https://docs.bitnami.com/aws/apps/wordpress/configuration/configure-phpfpm-processes/
http://myshell.co.uk/blog/2012/07/adjusting-child-processes-for-php-fpm-nginx/

Happy to help!


Was my answer helpful? Click on :heart:

hey,

we decided to teak our msql because this is the point where our problems starts.

Can you say to me why the msql server do not start again if we change this settings:

tmp_table_size
max_connections
table_cache
thread_concurrency
query_cache_limit
query_cache_size

We already tested our config on another server, so that we are wodnering whats going on here.

Do we need to do something special on bitnami lightsail?

@jota

are this settings blocked by bitnami?

Have you checked the log file of the database (/opt/bitnami/mysql/data/mysqld.log)to see if there is any error there?

No, we do not block any configuration in the server.

@jota Urgent :frowning: we finally created a Snapshot from this instance to see whats going on without take risk on our live instance.

Now we are facing the following issue, which is may the same: Our Mysql server of the snapshot cant be started.

Mysql Logs:
empty

Sys logs:
Jun 22 15:54:57 ip-172-26-18-186 systemd[1]: Failed to start LSB: bitnami init script.
Jun 22 15:54:57 ip-172-26-18-186 systemd[1]: bitnami.service: Unit entered failed state.
Jun 22 15:54:57 ip-172-26-18-186 systemd[1]: bitnami.service: Failed with result ‘exit-code’.

What we checked:

  • We are not running out of space:

    root@ip:/home/bitnami# sudo du -ch --max-depth=1 /opt/bitnami
    52K/opt/bitnami/config
    32M/opt/bitnami/letsencrypt
    116K/opt/bitnami/scripts
    2.1G/opt/bitnami/apache2
    3.0M/opt/bitnami/stats
    26M/opt/bitnami/bncert
    30M/opt/bitnami/bnsupport
    91G/opt/bitnami/apps
    4.2M/opt/bitnami/bnhelper
    3.6M/opt/bitnami/varnish
    3.5M/opt/bitnami/sqlite
    81M/opt/bitnami/php
    9.9G/opt/bitnami/mysql
    1.6M/opt/bitnami/licenses
    8.0K/opt/bitnami/img
    104K/opt/bitnami/var
    145M/opt/bitnami/common
    103G/opt/bitnami
    103Gtotal
    (320 GB SSD lightsail)

  • There is no other apache blocking us:

root@:/opt/bitnami# ps aux | grep 'apache\|mysql\|nginx\|/opt/bitnami'
root      2587  0.0  0.0 303028 10700 ?        Ss   16:13   0:00 php-fpm: master process (
/opt/bitnami/php/etc/php-fpm.conf)
root      2603  0.0  0.3 207096 56256 ?        Ss   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2645  0.0  0.0 815704 14260 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2646  0.0  0.0 815704 14272 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2647  0.0  0.0 815704 14332 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2648  0.0  0.0 815704 14268 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2649  0.0  0.0 815704 14268 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2650  0.0  0.0 815704 14340 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2651  0.0  0.1 1208920 18384 ?       Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2652  0.0  0.0 815704 14272 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2653  0.0  0.0 815704 14336 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2654  0.0  0.0 815704 14272 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2655  0.0  0.1 1077848 17276 ?       Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2656  0.0  0.1 1143384 17368 ?       Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2657  0.0  0.1 946776 17852 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2658  0.0  0.0 815704 14272 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2659  0.0  0.0 815704 14356 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
daemon    2660  0.0  0.0 815704 14296 ?        Sl   16:13   0:00 /opt/bitnami/apache2/bin/
httpd.bin -f /opt/bitnami/apache2/conf/httpd.conf -DDISABLE_BANNER
root      6127  0.0  0.0  14352   968 pts/0    S+   16:25   0:00 grep --color=auto apache\
|mysql\|nginx\|/opt/bitnami
  • Memory free:
  total        used        free      shared  buff/cache   available

Mem: 15G 225M 12G 56M 2.6G 15G
Swap: 0B 0B 0B

sudo systemctl status bitnami.service:

Loaded: loaded (/etc/init.d/bitnami; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-06-22 16:13:23 UTC; 46min ago
Process: 1312 ExecStart=/etc/init.d/bitnami start (code=exited, status=3)
Tasks: 1079
Memory: 212.7M
CPU: 12.114s
CGroup: /system.slice/bitnami.service
├─2587 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf)
├─2588 php-fpm: pool wordpress
├─2589 php-fpm: pool wordpress
├─2590 php-fpm: pool wordpress
├─2591 php-fpm: pool wordpress
├─2592 php-fpm: pool wordpress
├─2593 php-fpm: pool wordpress
├─2603 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2645 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2646 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2647 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2648 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2649 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2650 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2651 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2652 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2653 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2654 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2655 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2656 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2657 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2658 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2659 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
├─2660 /opt/bitnami/apache2/bin/httpd.bin -f /opt/bitnami/apache2/conf/httpd.co
└─3759 /usr/bin/gonit

Can you please help us to identify, why we cant start the mysql?

Support Ticket: Cant be uploaded (see private message)

I think it is another problem and not regarding to the CPU Burst mode. So please close this ticket.

New Ticket: (!) MySQL Server not Starting

We really need your help!

Hi @eSpox,

We will continue the conversation in that other thread.

Thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.