Magento 2.4.2 - consumed RAM-stop server

Keywords: Magento - AWS - Technical issue - Other
Description:
Hi,
on Bitnami cloud Hosting I have an instance of AWS R2 Large Server (10 ECUs, 16.0GiB RAM) and Magento 2.4.2.

I have running services: Apache, Elasticsearch, FPM, MariaDB, Memcached (inactive).

I did not change the default Magento stack settings. I don’t have any major traffic from a particular IP address (cd /opt/bitnami/apache2/logs/
tail -n 10000 access_log | awk ‘{print $ 1}’ | sort | uniq -c | sort -nr | head -n 10) - average 350-500 per IP.

Gradually, all the RAM is consumed and the server finally stops. (It depends on the operation, but on average after 24 hours the server will stop due to RAM consumption).

Do you have a solution to the problem of how to prevent this?

Hi @eduard,

We have a Support Tool that will gather relevant information for us to analyze your configuration and logs. Could you please execute it on the machine where the stack is running by following the steps described in the guide below? It’d be great if you can run it when you find that the services are stopped due to RAM consumption.

Please note that you need to paste the code ID that is shown at the end.

Hi,

here is the code: 8f01f773-5fb4-252f-9ce1-d82bcca5b18b

I also made this fix yesterday in /opt/bitnami/php/etc/memory.conf:

Instead of:

pm.max_children = 200
pm.start_servers = 130
pm.min_spare_servers = 130
pm.max_spare_servers = 150
pm.max_requests = 5000

I gave:

pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
pm.max_requests = 200

and at the same time in /opt/bitnami/php/etc/php-fpm.conf:

I set: (originally it was disable “;”)

emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 10s

Nothing is restarted yet and the server is running.

If you find a solution, I’ll be happy.

Regards

Eduard

Hi @eduard,

I took a look at the Bitnami Support tool information and I could see that you have plenty RAM memory available at that time

-----------------------------------
Display amount of free and used memory in the system
-----------------------------------
Running: free -m
In: /opt/bitnami

Output:

              total        used        free      shared  buff/cache   available
Mem:          15749        7856        3833         341        4059        7247
Swap:           634           0         634

there are only 4 PHP-FPM processes

daemon   19752  1.2  1.0 773504 172996 ?       S    07:47   0:51 php-fpm: pool www
root     25724  0.0  0.2 641592 42668 ?        Ss   May20   0:00 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf)
daemon   27425  1.0  1.0 755928 173548 ?       S    07:50   0:43 php-fpm: pool www
daemon   27431  1.0  1.2 833904 196124 ?       Sl   07:50   0:42 php-fpm: pool www

and the CPU usage looks fine too

-----------------------------------                                                                                                                                                                                                                           Check CPU usage for current processes                                                                                                                                                                                                                         -----------------------------------                                                                                                                                                                                                                           Running: ps -e -o pcpu,nice,state,cputime,args --sort -pcpu | awk '{print $1,$2,$3,$4,$5}'
In: /opt/bitnami

Output:

%CPU NI S TIME COMMAND
2.9 0 S 00:00:11 /opt/bitnami/bnsupport-tool
1.2 0 S 00:00:51 php-fpm:
1.0 0 S 00:00:43 php-fpm:
1.0 0 S 00:00:42 php-fpm:
...

Can you take a look at the PHP-FPM’s log file to see if there is any relevant info there?

tail -n30 /opt/bitnami/php/logs/php-fpm.log

I can see that there are at least 10 IPs with more than 1000 requests

-----------------------------------
Check performance issues: Count number of requests for the 10 most active IP addresses in the last 100.000 requests
-----------------------------------
Running: tail -n 100000 access_log | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10 | awk '{print $1}'
In: /opt/bitnami/apache2/logs/

Output:

5941
2063
1553
1318
1267
1187
1150
1057
1045
1043

Hi,
since the last setting, my free memory has dropped to 1G.

The php-fpm.log shows me: “server reached pm.max_children setting (5), consider raising it.”

I changed the values for php-fpm in memory.conf to

pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 20
pm.max_requests = 500

If I use the command:

ps --no-headers -o "rss, cmd" -C php-fpm | awk '{sum + = $ 1} END {printf ("% d% s \ n", sum / NR / 1024, "Mb")}'

I get the average of one php-fpm process: 127Mb

The value pm.max_children = 50 should be within the limit for setting max_children.

I will watch how long my memory drops below 1G again. Shouldn’t fpm restart automatically? Or the solution would be regular e.g. about 2 AM restart server? Do you have a solution for this?

Regards.

Eduard

Hi @eduard,

If PHP-FPM is reaching the limits in regards to number of connections, it won’t be restarted, it’ll simply stop accepting connections.

This shouldn’t be the solution. As I mentioned in my previous post, it seems that there were 10 IPs with more than 1000 requests, did you take a look at that?

Hi,
I saw traffic:

and I have :
2203
884
702
674
539
450
413
382
246
239

Only one IP did not have requests for specific products (884 requests - I need to adjust security), but the others were requests for specific products, which also ended up creating an order.

An IP with requests (2203) is a shared Internet, and more users have this IP, and various customers have also placed an order from it. I don’t think I have any robots to go to the web right now.

Nevertheless, the RAM has already been reduced to 1.5G.
I don’t know where to look for the problem.

Regards

Eduard

Hi @eduard,

It seems your server is receiving many requests and the access_log file is being rotated due to the large amount of data it contains

apache/logs:
total 53080
drwxrwxr-x  2 root root     4096 May 16 00:00 .
drwxr-xr-x 18 root root     4096 Apr 28 08:57 ..
-rw-r--r--  1 root root 50565048 May 21 08:50 access_log
-rw-r--r--  1 root root   591585 May  5 21:16 access_log-20210507.gz
-rw-r--r--  1 root root  1468504 May 13 23:59 access_log-20210514.gz
-rw-r--r--  1 root root  1623082 May 15 23:59 access_log-20210516.gz

You can extract any of those files and run the same commands using the access_log file you extracted (instead of the one that Apache is using right now) to see if you find new information there.

Do you continue getting this error in the php-fpm.log file?

Hi,

I’ll also look at these logs to see if I can find anything there.

I restarted the whole server, I am adjusting the security to limit the operation.

The error “server reached pm.max_children setting (5), consider raising it” no longer appears after increasing the values.

Regards

Eduard

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.