Site is down because of too many dispatching Errors: AH01075: Error dispatching request to : (polling)

guangbin 2021-01-31 07:02:06 UTC #1

Keywords: General - AWS - Technical issue - Other
Description:

I have a site with very little traffic hosted on AWS Ligthsail for around 2 years. It worked pretty well until recently it is down for a few times. I have to reboot the server to recover it. But after a few hours it is down again so I have reboot it again.

When I checked the log file I found there are a lot errors as the following in the /opt/bitnami/apache2/logs/error_log

[Sun Jan 31 02:08:40.762173 2021] [proxy_fcgi:error] [pid 6570:tid 139928482440960] (70007)The timeout specified has expired: [client 162.158.165.149:56152] AH01075: Error dispatching request to : (polling)  
[Sun Jan 31 02:09:09.522597 2021] [proxy_fcgi:error] [pid 6570:tid 139928925964032] (70007)The timeout specified has expired: [client 172.69.134.178:32252] AH01075: Error dispatching request to : (polling)  
[Sun Jan 31 02:09:40.783398 2021] [proxy_fcgi:error] [pid 6570:tid 139928398513920] (70007)The timeout specified has expired: [client 162.158.165.23:52752] AH01075: Error dispatching request to : (polling)  
[Sun Jan 31 02:10:08.386622 2021] [proxy_fcgi:error] [pid 6570:tid 139928390121216] (70007)The timeout specified has expired: [client 172.69.134.112:41172] AH01075: Error dispatching request to : (polling)  
[Sun Jan 31 02:10:32.002887 2021] [proxy_fcgi:error] [pid 6570:tid 139928574760704] (70007)The timeout specified has expired: [client 108.162.237.57:51154] AH01075: Error dispatching request to : (polling)  
[Sun Jan 31 02:10:39.382545 2021] [proxy_fcgi:error] [pid 6570:tid 139928557975296] (70007)The timeout specified has expired: [client 162.158.166.238:17712] AH01075: Error dispatching request to : (polling)  
[Sun Jan 31 02:11:09.674597 2021] [proxy_fcgi:error] [pid 6570:tid 139928440477440] (70007)The timeout specified has expired: [client 162.158.167.63:53314] AH01075: Error dispatching request to : (polling)  
[Sun Jan 31 02:11:35.986611 2021] [proxy_fcgi:error] [pid 6570:tid 139928541189888] (70007)The timeout specified has expired: [client 172.69.170.127:52568] AH01075: Error dispatching request to : (polling), referer: https://xxxxxxxxx.com
[Sun Jan 31 02:11:40.693157 2021] [proxy_fcgi:error] [pid 4087:tid 139928951142144] (70007)The timeout specified has expired: [client 172.68.146.189:24666] AH01075: Error dispatching request to : (polling)  
[Sun Jan 31 03:03:21.958415 2021] [mpm_event:error] [pid 4077:tid 139929162622720] AH10159: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting

So I edit the /opt/bitnami/apache2/confhttpd.conf file and add the following, but it is of no use.

<IfModule mpm_event_module>   
    StartServers 3   
    ServerLimit 16   
    MinSpareThreads 75   
    MaxSpareThreads 250   
    ThreadsPerChild 25   
    MaxRequestWorkers 400   
    MaxConnectionsPerChild 1000   
</IfModule>

So after that I was thinking about been attacked by bot/attacker. I used the following command to check my access_log to find any suspicious activity

tail -n 100000 access_log | awk '{print $1}' | sort | uniq -c| sort -nr | head -n 10

And I found result as the following. The IP of 172.26.14.163 is my current instance IP. And it seems that the rest IPs did not request my server heavily. Despite this, I still blocked the first 3 IPs. But it still is of no use util now.

22767 172.26.14.163  
    518 162.158.165.23  
    501 172.69.170.127  
    478 172.69.170.145  
    468 172.69.170.147  
    406 3.91.235.204  
    401 172.69.170.101   
    399 162.158.166.240  
    388 172.69.135.17  
    335 162.158.166.92

I have checked the Access_log last year as well. And the result is almost the same as the above. And the server was not down at that time. So I guess there are no bots/attackers.

I have almost read all the posts related to this issue and done everything I can consider. The problem has not been solved.Could anybody help me. Thanks a lot.

fdepaz 2021-02-01 15:58:14 UTC #2

Hello @guangbin,

We have a Support Tool that will gather relevant information for us to analyze your configuration and logs. Could you please execute it on the machine where the stack is running by following the steps described in the guide below?

How to Run the Bitnami Support Tool in a cloud image or virtual machine

Please note that you need to paste the code ID that is shown at the end.

guangbin 2021-02-03 04:01:53 UTC #3

Hi fdepaz.
Thank you very much for your reply.
I used the Bitnami Support Tool and created a zip file and send to you manually through google drive. I have sent you a private message about the link to the google drive.

The following is the update of the issue
After I posted here in the past weekend, when I check the php log at /opt/bitnami/php/var/log, I found that there are error saying that
server reached pm.max_children setting (5), consider raising it

So I edit the php config file at /opt/bitnami/php/etc/php-fpm.conf and modified all the default value to the following

    pm.max_children = 30
    pm.start_servers = 10
    pm.min_spare_servers=10
    pm.max_spare_servers=24
    pm.max_requests = 500

and restarted php-fpm

After this modification, it worked for around 2 days. But the server is down again this morning as in last week. I checked the apache error_log. It is the same error as in last week.

Website running slow

fdepaz 2021-02-03 17:30:40 UTC #4

Hello @guangbin,

There is some information missing in the bnsupport file, mainly related to free space and memory. Could you share the output of the following commands:

-----------------------------------
Check memory usage for all current processes
-----------------------------------
$ ps -e -orss=,args= | sort -b -k1,1n | awk '{print $1,$2}'

-----------------------------------
Check CPU usage for current processes
-----------------------------------
$ ps -e -o pcpu,nice,state,cputime,args --sort -pcpu | awk '{print $1,$2,$3,$4,$5}'

-----------------------------------
Display amount of free and used memory in the system
-----------------------------------
$ free -m

guangbin 2021-02-04 03:02:29 UTC #5

Hi, fdepaz

Thank you for your reply.
I have executed the commands and sent back the result to you in a private message.

fdepaz 2021-02-04 16:48:17 UTC #6

Hello @guangbin,

I'm seeing very little memory free, which may be causing performance issues and resulting in timeout errors. As your application has been running for 2 years, this performance issue could also be related to MySQL binlogs files getting out of hand. Could you share the output of the following commands:

-----------------------------------
Find biggest directories in installdir
-----------------------------------
$ du -h . -d 1 | sort -h
In: /opt/bitnami

$ du -hs /opt/bitnami/mysql/data

guangbin 2021-02-05 08:06:18 UTC #7

The following is the result of your command. It seems that there is no storage issue.
As I said, I have changed the value of pm.max_children in php-fpm.conf from 5 to 30. That could be the reason for little memory free.

bitnami@ip-xx.xx.xx.xx:/opt/bitnami$ sudo du -h . -d 1 | sort -h
8.0K ./img
44K ./docs
52K ./config
96K ./var
116K ./scripts
908K ./licenses
2.8M ./varnish
3.0M ./stats
3.3M ./sqlite
26M ./bncert
32M ./letsencrypt
42M ./apps
45M ./git
58M ./bnsupport
136M ./common
175M ./php
241M ./frameworks
488M ./mysql
535M ./apache2
1.8G .

bitnami@ip-xx.xx.xx.xx:/opt/bitnami$ sudo du -hs /opt/bitnami/mysql/data
200M /opt/bitnami/mysql/data

fdepaz 2021-02-05 12:32:24 UTC #8

Hello @guangbin,

In that case, the most probable cause is the memory's limitation. Even if the pm.max_children parameter is increased, there is little memory to work with. The minimum recommended memory size for LAMP is 600MB so I would recommend increasing the instance size to 1GB of RAM and checking whether that solves the issue. If it is such a case we may need to update our documentation.

guangbin 2021-02-08 03:14:02 UTC #9

Hi. @fdepaz
OK. It has worked well for 5 days since my last reboot. I will try to increase the memory if it crash again and give you feedback. Thank you very much for your suggestion.

system 2021-02-22 03:14:08 UTC #10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!