Mod_security3 memory leak

jlago3577 2021-02-01 23:22:17 UTC #1

Keywords: PrestaShop - Google Cloud Platform - Technical issue - Other
bnsupport ID: 51f92eb1-b704-e907-84b8-c16292868531
Description:
I'm facing an issue with mod_security where after I enable it, my server's memory starts slowly climbing up to 100%. Maybe there's a memory leak on the module? I'm using the latest 1.7.7.1-0 version of the gcp stack.

jlago3577 2021-02-01 23:24:22 UTC #2

Right after I enable it, memory goes from 40% usage to 80% in around 24 hours.

fdepaz 2021-02-02 10:01:54 UTC #3

Hello @jlago3577,

This may be related to a heavy workload of the module. I have seen that you are getting a lot of requests from a single IP, could you check if that is an expected behaviour? Please follow the guide Deny Connections From Bots/Attackers Using Apache.

jlago3577 2021-02-02 13:32:51 UTC #4

Hello!

How do I block IPV6 addresses on the varnish vcl? Do I enter them like so:

acl forbidden {
     "192.0.2.235";
     "2001:db8:85a3:8d3:1319:8a2e:370:7348";
}

Thanks.

jlago3577 2021-02-02 14:12:49 UTC #5

Also, after restarting apache the memory usage goes back to normal.

jlago3577 2021-02-02 22:51:13 UTC #6

I can't see anything serious. The ip with the most requests is the admin.

I've restarted apache 10 hours ago, and it has slowly climbed from 30% to 60% memory usage, and it's continuing.

This only happens with mod_security3 on.

I can see from the mod_security github there has been a memory leak fix on the latest 3.0.4 release, and a memory leak fix on the last commit to master https://github.com/SpiderLabs/ModSecurity/commit/6ca028b6f5713ea505bcdd39a43a1aaa4fba936e
How would I go about making and installing the latest release on this stack (Approach A)?

fdepaz 2021-02-03 12:10:01 UTC #7

Hello @jlago3577,

Following Varnish documentation, you can block IPs with the following blocks:

acl unwanted {
    "localhost";
    "192.168.1.0"/24; /* and everyone on the local network */
}

sub vcl_recv {
  if (client.ip ~ unwanted) {
      return(synth(403, "Access denied."));
  }
}

Varnish 6.0 supports IPv6, so there shouldn't be any problems.

Regarding the possible cause, it may be the recently fixed memory leak you shared, but we will need to wait until that fix is included in a new modSecurity release for our internal tools to automatically update to the latest version. The version 3.0.4 was released more than a year ago, so the new fix is yet to be included.

jlago3577 2021-02-10 21:23:55 UTC #8

Guys, I think this should really be fixed. The module is unusable, and the last mod_security release was a year ago.
There must be some way to compile and install, like it's done with mod_evasive: https://docs.bitnami.com/aws/infrastructure/lamp/configuration/enable-modules/#mod_evasive

fdepaz 2021-02-11 15:43:51 UTC #9

Hello @jlago3577,

I will work on reproducing this and monitoring memory usage and get back to you. Could you confirm you followed our Mod_security section at Enable Different Apache Modules to enable it?

jlago3577 2021-02-11 16:22:06 UTC #10

Yes. I've followed that exact tutorial to enable mod security.

fdepaz 2021-02-24 11:43:23 UTC #11

Hello @jlago3577,

Sorry for the late response, I have tried to reproduced the issue using different configurations but I'm not seeing any red flags in the memory management. Using a barebones Prestashop installation with mod_security enabled I have observed that memory usage has increased over time as I used the site, but the memory was freed when the available quantity got low enough:

fdepaz@bitnami-prestashop-08ec:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:           1693         400         377          26         916        1102
Swap:             0           0           0
...
fdepaz@bitnami-prestashop-08ec:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:           1693         473         301          27         919        1031
Swap:             0           0           0
fdepaz@bitnami-prestashop-08ec:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:           1693         484         289          28         919        1016
Swap:             0           0           0
...
fdepaz@bitnami-prestashop-08ec:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:           1693         637          73          63         982         826
Swap:             0           0           0
fdepaz@bitnami-prestashop-08ec:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:           1693         701          91          63         900         762
Swap:             0           0           0
fdepaz@bitnami-prestashop-08ec:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:           1693         707          84          63         902         756
Swap:             0           0           0
fdepaz@bitnami-prestashop-08ec:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:           1693         690         100          63         902         773
Swap:             0           0           0

Does this behavior correspond to your case? Have you seen any memory increase that spiked and compromised your site?

Regards,
Francisco de Paz

jlago3577 2021-02-24 15:32:21 UTC #12

Does this behavior correspond to your case?

Yes. Memory starts getting filled up the more the website is used.

Have you seen any memory increase that spiked and compromised your site?

There has been a memory increase that crashed my server.

Below is the memory utilization of my server during and after mod_security3 being turned on.

fdepaz 2021-02-25 12:46:53 UTC #13

Hello @jlago3577,

Could you confirm there is not any suspicious behaviour in apache's access_log during the time the module was enabled. My testing environment was a new Prestashop instance so maybe there are exist some incompatibilities between mod_security and a plugin or theme. Can you please try to reproduce your server crash in a fresh instance? If the problem comes indeed from mod_security running by itself we will make an in-depth investigation internally to address this issue.

Best regards,
Francisco de Paz

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!