Server freezes every now and then

Keywords: LAMP/MAMP/WAMP - AWS - Technical issue - Services (Apache, MariaDB, MySQL…)

bnsupport ID: 4db1e88d-9e94-4ff6-cd26-c97fa804a2d8

bndiagnostic output:

? Apache: Found possible issues
? Mariadb: Found possible issues
? Resources: Found possible issues
https://docs.bitnami.com/general/apps/wordpress/troubleshooting/debug-errors-apache/
https://docs.bitnami.com/bch/apps/moodle/troubleshooting/deny-connections-bots-apache/
https://docs.bitnami.com/aws/apps/wordpress/troubleshooting/debug-errors-mariadb/
https://docs.bitnami.com/installer/faq/linux-faq/administration/increase-memory-linux/

bndiagnostic failure reason: I do not know how to perform the changes explained in the documentation

Description:
Hello,

my LAMP freezes/is down every now and then. The CPU usage is 50% and then the server is no longer responsive for 3-30 minutes. This happens at any given times. The website was running on a Digital Ocean $5 test server before without any issue but since we moved to AWS T3.Small the server performance has dropped drastically. I am not sure what causes the problem?

Hello @Mad2kx,

The bnsupport ID you shared corresponds to a WordPress stack instead of a LAMP one, is this the correct one?

Regarding your instance, a high number of incoming requests originate from one or more unique IP addresses. This could indicate a bot attack. The following guide shows how to check for and block suspicious IP addresses.

https://docs.bitnami.com/bch/apps/moodle/troubleshooting/deny-connections-bots-apache/

Apart from that, there aren’t any red flags in the logs, with timeouts being the only issue. It seems you are running a bit low on memory:

-----------------------------------
Display amount of free and used memory in the system
-----------------------------------
Running: free -m
In: /opt/bitnami

Output:

              total        used        free      shared  buff/cache   available
Mem:           1959        1364         114         142         481         272
Swap:             0           0           0

I would suggest increasing your instance size to see if the issue is coming from insufficient resources:

https://docs.bitnami.com/aws/faq/administration/change-server-type/

Did you recall changing the theme, plugins, or general configuration compared to your previous instance that could explain the performance drop?

Regards,
Francisco de Paz

Hi @fdepaz thanks for the response. Indeed it is a Wordpress and not Lamp, apologies. So since we migrated to AWS we did not change anything. On DigitalOcean it was just a staging site and we copied over the exact same website. I suspect there’s a bot attack happening here and recommended the client to connect to Cloudflare. If this does not help I might increase the AWS Instance. The interesting part is that when the website is down I log into AWS and reboot the instance twice and it come sback online. If I do not make another reboot via Terminal afterwards the website is offline a few hours later. So only if I do a third reboot via Terminal I have peace for 1-2 days.

Hello @Mad2kx,

That is strange behavior, does it matter if you do the third restart after a while has passed or do you always do it after restarting it from AWS console? Doesn’t the instance come back after a single AWS restart? Have you been able to restart it from the get-go?

If you can, please share the running processes of the instance when the issue appears and/or before making the restart from the console:

ps aux

Regards,
Francisco de Paz

Hi @fdepaz,

Thanks for your response.

About 1h ago the instance went offline and I tried logging in via Terminal but it was not responding. I then had to reboot via AWS 3 times until it came back online. I was then able to login via ssh Terminal and did a 4th reboot via terminal (sudo reboot) and then the instance was finally back up and running.

Running the command gives me the following output

    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root         1  0.0  0.4  21916  9776 ?        Ss   10:26   0:01 /sbin/init
    root         2  0.0  0.0      0     0 ?        S    10:26   0:00 [kthreadd]
    root         3  0.0  0.0      0     0 ?        I<   10:26   0:00 [rcu_gp]
    root         4  0.0  0.0      0     0 ?        I<   10:26   0:00 [rcu_par_gp]
    root         6  0.0  0.0      0     0 ?        I<   10:26   0:00 [kworker/0:0H-k
    root         8  0.0  0.0      0     0 ?        I<   10:26   0:00 [mm_percpu_wq]
    root         9  0.0  0.0      0     0 ?        S    10:26   0:00 [ksoftirqd/0]
    root        10  0.0  0.0      0     0 ?        I    10:26   0:00 [rcu_sched]
    root        11  0.0  0.0      0     0 ?        I    10:26   0:00 [rcu_bh]
    root        12  0.0  0.0      0     0 ?        S    10:26   0:00 [migration/0]
    root        13  0.0  0.0      0     0 ?        I    10:26   0:00 [kworker/0:1-mm
    root        14  0.0  0.0      0     0 ?        S    10:26   0:00 [cpuhp/0]
    root        15  0.0  0.0      0     0 ?        S    10:26   0:00 [cpuhp/1]
    root        16  0.0  0.0      0     0 ?        S    10:26   0:00 [migration/1]
    root        17  0.0  0.0      0     0 ?        S    10:26   0:00 [ksoftirqd/1]
    root        19  0.0  0.0      0     0 ?        I<   10:26   0:00 [kworker/1:0H-k
    root        20  0.0  0.0      0     0 ?        S    10:26   0:00 [kdevtmpfs]
    root        21  0.0  0.0      0     0 ?        I<   10:26   0:00 [netns]
    root        22  0.0  0.0      0     0 ?        S    10:26   0:00 [kauditd]
    root        23  0.0  0.0      0     0 ?        I    10:26   0:00 [kworker/1:1-ev
    root        24  0.0  0.0      0     0 ?        S    10:26   0:00 [khungtaskd]
    root        25  0.0  0.0      0     0 ?        S    10:26   0:00 [oom_reaper]
    root        26  0.0  0.0      0     0 ?        I<   10:26   0:00 [writeback]
    root        27  0.0  0.0      0     0 ?        S    10:26   0:00 [kcompactd0]
    root        28  0.0  0.0      0     0 ?        SN   10:26   0:00 [ksmd]
    root        29  0.0  0.0      0     0 ?        SN   10:26   0:00 [khugepaged]
    root        30  0.0  0.0      0     0 ?        I<   10:26   0:00 [crypto]
    root        31  0.0  0.0      0     0 ?        I<   10:26   0:00 [kintegrityd]
    root        32  0.0  0.0      0     0 ?        I<   10:26   0:00 [kblockd]
    root        33  0.0  0.0      0     0 ?        S    10:26   0:00 [watchdogd]
    root        34  0.0  0.0      0     0 ?        I    10:26   0:00 [kworker/u4:1-e
    root        36  0.0  0.0      0     0 ?        S    10:26   0:00 [kswapd0]
    root        52  0.0  0.0      0     0 ?        I<   10:26   0:00 [kthrotld]
    root        53  0.0  0.0      0     0 ?        I<   10:26   0:00 [ipv6_addrconf]
    root        62  0.0  0.0      0     0 ?        I<   10:26   0:00 [kstrp]
    root        67  0.0  0.0      0     0 ?        I    10:26   0:00 [kworker/1:2-ev
    root       105  0.0  0.0      0     0 ?        I<   10:26   0:00 [nvme-wq]
    root       107  0.0  0.0      0     0 ?        I<   10:26   0:00 [nvme-reset-wq]
    root       108  0.0  0.0      0     0 ?        I<   10:26   0:00 [nvme-delete-wq
    root       109  0.0  0.0      0     0 ?        I    10:26   0:00 [kworker/u4:2-e
    root       110  0.0  0.0      0     0 ?        I<   10:26   0:00 [ena]
    root       174  0.0  0.0      0     0 ?        I<   10:26   0:00 [kworker/u5:0]
    root       176  0.0  0.0      0     0 ?        S    10:26   0:00 [jbd2/nvme0n1p1
    root       177  0.0  0.0      0     0 ?        I<   10:26   0:00 [ext4-rsv-conve
    root       236  0.0  0.3  23880  7664 ?        Ss   10:26   0:00 /lib/systemd/sy
    root       249  0.0  0.2  20068  4840 ?        Ss   10:26   0:00 /lib/systemd/sy
    root       304  0.0  0.2   8084  4836 ?        Ss   10:26   0:00 /usr/sbin/haveg
    root       329  0.0  0.0      0     0 ?        I<   10:26   0:00 [kworker/1:1H-k
    root       361  0.0  0.2   9488  5544 ?        Ss   10:26   0:00 /sbin/dhclient 
    root       469  0.0  0.2   9488  5496 ?        Ss   10:26   0:00 /sbin/dhclient 
    root       552  0.0  0.0      0     0 ?        I<   10:26   0:00 [kworker/0:1H-k
    root       554  0.0  0.3  19480  7304 ?        Ss   10:26   0:00 /lib/systemd/sy
    root       555  0.0  0.2 225824  5436 ?        Ssl  10:26   0:00 /usr/sbin/rsysl
    root       557  0.0  0.1   8476  2592 ?        Ss   10:26   0:00 /usr/sbin/cron 
    message+   560  0.0  0.1   8716  3388 ?        Ss   10:26   0:00 /usr/bin/dbus-d
    daemon     574  0.0  0.0   5488  1864 ?        Ss   10:26   0:00 /usr/sbin/atd -
    _chrony    584  0.0  0.1   4816  2124 ?        S    10:26   0:00 /usr/sbin/chron
    _chrony    585  0.0  0.0   4816   268 ?        S    10:26   0:00 /usr/sbin/chron
    root       589  0.0  0.8  29660 16976 ?        Ss   10:26   0:00 /usr/bin/python
    root       595  0.0  0.3  15852  6804 ?        Ss   10:26   0:00 /usr/sbin/sshd 
    root       687  0.0  1.9 313892 39132 ?        Ss   10:27   0:00 php-fpm: master
    daemon     713  0.2  9.1 420944 182768 ?       S    10:27   0:05 php-fpm: pool w
    daemon     714  0.3  9.3 423236 187140 ?       S    10:27   0:06 php-fpm: pool w
    daemon     715  0.2  9.1 423120 183548 ?       S    10:27   0:06 php-fpm: pool w
    daemon     716  0.2  9.6 431400 192996 ?       S    10:27   0:05 php-fpm: pool w
    daemon     717  0.2  8.1 408168 163096 ?       S    10:27   0:05 php-fpm: pool w
    daemon     718  0.2  8.8 423572 177732 ?       S    10:27   0:05 php-fpm: pool w
    daemon     719  0.3  9.9 437348 198948 ?       S    10:27   0:06 php-fpm: pool w
    daemon     720  0.2  8.6 412348 173632 ?       S    10:27   0:05 php-fpm: pool w
    daemon     721  0.1  8.7 414364 174612 ?       S    10:27   0:03 php-fpm: pool w
    daemon     723  0.2  8.7 422564 176356 ?       S    10:27   0:04 php-fpm: pool w
    daemon     724  0.2  8.1 408532 164212 ?       S    10:27   0:05 php-fpm: pool w
    daemon     725  0.2  9.4 427132 189044 ?       S    10:27   0:05 php-fpm: pool w
    daemon     727  0.2  8.7 420940 174848 ?       S    10:27   0:04 php-fpm: pool w
    daemon     728  0.2  9.6 432496 193944 ?       S    10:27   0:05 php-fpm: pool w
    daemon     729  0.3  8.7 421640 175524 ?       S    10:27   0:06 php-fpm: pool w
    daemon     730  0.2  8.5 410096 171260 ?       S    10:27   0:04 php-fpm: pool w
    daemon     731  0.2  8.3 412396 167032 ?       S    10:27   0:05 php-fpm: pool w
    root       790  0.0  0.4  14588  8304 ?        Ss   10:27   0:00 /opt/bitnami/ap
    daemon     793  0.1  1.0 2326640 21584 ?       Sl   10:27   0:02 /opt/bitnami/ap
    daemon     794  0.1  1.3 2326960 26372 ?       Sl   10:27   0:02 /opt/bitnami/ap
    daemon     795  0.1  1.1 2326704 23540 ?       Sl   10:27   0:03 /opt/bitnami/ap
    mysql     1293  0.7 11.1 1248932 223296 ?      Sl   10:27   0:16 /opt/bitnami/ma
    root      1349  0.0  0.7 1155220 14820 ?       Ssl  10:27   0:00 /opt/bitnami/go
    daemon    1405  0.1  1.3 2326668 28056 ?       Sl   10:27   0:03 /opt/bitnami/ap
    daemon    1569  0.2  9.6 426700 194348 ?       S    10:27   0:04 php-fpm: pool w
    daemon    1571  0.2  8.8 414840 176656 ?       S    10:27   0:06 php-fpm: pool w
    daemon    1576  0.2  8.7 413044 176032 ?       S    10:27   0:04 php-fpm: pool w
    root      1704  0.0  0.0      0     0 ?        I    10:41   0:00 [kworker/0:0-cg
    daemon    1793  0.1  9.2 432320 185448 ?       S    10:53   0:01 php-fpm: pool w
    daemon    1878  0.0  0.5 313892 10848 ?        S    10:59   0:00 php-fpm: pool w
    root      1890  0.0  0.3  16600  7896 ?        Ss   11:02   0:00 sshd: bitnami [
    root      1893  0.0  0.0      0     0 ?        I    11:02   0:00 [kworker/1:0-me
    bitnami   1894  0.3  0.4  21028  8332 ?        Ss   11:02   0:00 /lib/systemd/sy
    bitnami   1895  0.0  0.1  22880  2244 ?        S    11:02   0:00 (sd-pam)
    bitnami   1903  0.0  0.2  16600  4700 ?        R    11:02   0:00 sshd: bitnami@p
    bitnami   1904  0.6  0.2   7692  4500 pts/0    Ss   11:02   0:00 -bash
    bitnami   1909  0.0  0.1  10632  3112 pts/0    R+   11:02   0:00 ps aux

Hello @Mad2kx,

I don’t see any red flags in your running processes, though php-fpm is indeed using a lot of resources. I would first check the bot attack option given the number of request coming from a single IP. If that doesn’t solve the issue, I would experimentally increase the instance available memory as it is also a bit low on free memory.

Regards,
Francisco de Paz