How to cure hacked Wordpress Bitnami?

Keywords: WordPress - AWS - Technical issue - Permissions
Description:
My WP-Installation hosted on AWS Lightsail using Bitnami was obviously hacked as I learned from Google some days ago - they reported a hacked URL. I checked my database and found no damage there (as far as I was able to tell). But it seems the intruder has managed to get access to my WP-backend and inserted some files and directories with malicious code. Here is a list of my htdocs directory:

bitnami@ip-XXX-XXX-XXX-XXX:~/apps/wordpress/htdocs$ ls -a -l
total 320
drwxrwxr-x 24 bitnami daemon  4096 May 13 02:09 .
drwxrwxr-x  7 bitnami root    4096 Apr 27  2020 ..
drwxr-xr-x  2 daemon  daemon  4096 Apr 26 03:02 09h658h
drwxr-xr-x  2 daemon  daemon  4096 Apr 26 03:02 2mn9gu
-rwxr-xr-x  1 daemon  daemon  1797 May 22  2020 5a2q8ruqgw_index.php
drwxr-xr-x  2 daemon  daemon  4096 Apr 26 03:03 5gtcg
drwxr-xr-x  2 daemon  daemon  4096 Apr 26 03:03 5p3l5
drwxr-xr-x  2 daemon  daemon  4096 Apr 26 03:04 9ho5se
drwxr-xr-x  3 daemon  daemon  4096 Oct 15  2020 blog
drwxr-xr-x  2 daemon  daemon  4096 Apr 26 03:04 crawlspace-full-oxlu
drwxr-xr-x  2 daemon  daemon  4096 Apr 26 03:03 frontier-co-twlcn
-rw-rw-r--  1 bitnami daemon   976 Sep  9  2020 .htaccess
-rw-rw-r--  1 bitnami daemon   405 Sep  9  2020 index.php
drwxr-xr-x  3 daemon  daemon  4096 Apr 26 07:56 jgvg91
-rwxr-xr-x  1 daemon  daemon  1797 May 22  2020 juz4n9ms3d_index.php
drwxr-xr-x  2 daemon  daemon  4096 Apr 26 03:03 leaning-over-glhddb
-rw-rw-r--  1 bitnami daemon 19915 Sep  9  2020 license.txt
drwxr-xr-x  3 daemon  daemon  4096 Apr 26 12:20 lupine-flower-vnti
drwxr-xr-x  3 daemon  daemon  4096 Apr 26 09:46 m1tud
drwxr-xr-x  3 daemon  daemon  4096 Apr 26 12:22 mbr0bi18
drwxr-xr-x  2 daemon  daemon  4096 Apr 26 03:02 nxvw9dwi
drwxr-xr-x  2 daemon  daemon  4096 Apr 26 03:03 philosophical-thoughts-ddrxfm
drwxr-xr-x  2 daemon  daemon  4096 Sep  4  2020 .quarantine
-rw-rw-r--  1 bitnami daemon  7278 May 13 02:09 readme.html
drwxr-xr-x  3 daemon  daemon  4096 Oct 15  2020 site
drwxr-xr-x  3 daemon  daemon  4096 Apr 26 11:39 succinic-anhydride-lpzzcdd
-rwxr-xr-x  1 daemon  daemon  1797 May 22  2020 thfyqvgbrk_index.php
drwxrwxrwx  2 daemon  daemon  4096 Sep  4  2020 .tmb
-rw-rw-r--  1 bitnami daemon  7101 Sep  9  2020 wp-activate.php
drwxrwxr-x  9 bitnami daemon  4096 May 10 18:45 wp-admin
-rw-rw-r--  1 bitnami daemon   351 Apr 15  2020 wp-blog-header.php
-rw-rw-r--  1 bitnami daemon  2332 Sep  9  2020 wp-comments-post.php
-rw-rw-r--  1 bitnami daemon  5202 May 20  2020 wp-config.php
-rw-rw-r--  1 bitnami daemon  2913 Apr 15  2020 wp-config-sample.php
drwxrwxr-x 20 bitnami daemon  4096 Jul 15  2020 wp-content
-rw-rw-r--  1 bitnami daemon  3940 Apr 15  2020 wp-cron.php
drwxrwxr-x 24 bitnami daemon 12288 Apr 30 16:31 wp-includes
-rw-rw-r--  1 bitnami daemon  2496 Apr 15  2020 wp-links-opml.php
-rw-rw-r--  1 bitnami daemon  3300 Apr 15  2020 wp-load.php
-rw-rw-r--  1 bitnami daemon 48761 Sep  9  2020 wp-login.php
-rw-rw-r--  1 bitnami daemon  8509 Apr 29  2020 wp-mail.php
-rw-rw-r--  1 bitnami daemon 20181 Sep  9  2020 wp-settings.php
-rw-rw-r--  1 bitnami daemon 31159 Sep  9  2020 wp-signup.php
-rw-rw-r--  1 bitnami daemon  4755 Apr 15  2020 wp-trackback.php
-rw-rw-r--  1 bitnami daemon  3236 Sep  9  2020 xmlrpc.php
-rwxr-xr-x  1 daemon  daemon  1797 May 22  2020 zmnrmhiigz_index.php

How can I find out more information about the intruder or where else could he have inserted stuff? Also, why is he acting as daemon and not bitnami?

Is there any tool within bitnami that helps curing such a hack?

Thank you for any advice!

Hi @jonathan.rhein

Thanks for using Bitnami WordPress and sorry to hear that!

How can I find out more information about the intruder or where else could he have inserted stuff?

if the intruder has accessed your site using SSH, there is a system log that records event details related to authentication (like connections or executions using sudo) that you might find helpful:

$ sudo cat /var/log/auth.log
...
May 17 08:59:32 bitnami-wordpress-33f2 sshd[2932]: Accepted publickey for jcarmona from 7X.XXX.XX.XX port 40922 ssh2: ECDSA SHA256:4UMcjXXXXXXXXXXXXXXXXXXX2/I
May 17 08:59:32 bitnami-wordpress-33f2 sshd[2932]: pam_unix(sshd:session): session opened for user jcarmona by (uid=0)
...
May 17 09:03:10 bitnami-wordpress-33f2 sudo: jcarmona : TTY=pts/0 ; PWD=/home/jcarmona ; USER=root ; COMMAND=/usr/bin/cat /var/log/auth.log
May 17 09:03:10 bitnami-wordpress-33f2 sudo: pam_unix(sudo:session): session opened for user root by jcarmona(uid=0)

where else could he have inserted stuff

About the affected files, I would say that the safest option for you would be to perform a complete restoration using a pre-existing backup. There is no easy way that comes to my mind in other to tell them apart. Additionally, bear in mind that not only may they have inserted files, but altered the existing ones too.

Also, why is he acting as daemon and not bitnami?

The daemon user is the one that the Apache server uses and has more limitations on what it can do than bitnami (it is a security feature we implement).

I can’t tell you the exact reason they are using daemon, but it can totally be related to the fact that they could not impersonate the bitnami user and had too proceed using daemon (which, as mentioned before, is more restrictive).

Is there any tool within bitnami that helps curing such a hack?

Unfortunately, no. In my humble opinion, there is no specific need for this to happen. If you perform regular backups of your site and keep a good security policy, you could revert to a previous state even in the case something like this happens. In addition, you should keep your instance up-to-date so as to run the latest security patches.

We have some guides that cover this topic and that you might find interesting:
https://docs.bitnami.com/aws/apps/wordpress/troubleshooting/enforce-security/
https://docs.bitnami.com/aws/apps/wordpress/troubleshooting/deny-connections-bots-apache/

Best regards,
Jose Antonio Carmona


Was my answer helpful? Click on :heart:

1 Like

Hi @jcarmona, thank you so much for your detailed and very helpful answer. I checked the auth.log and it seems that no one managed to get access via SSH. At least there is no log starting like "Accepted publickey for…” (like you posted) on the respective days when the malicious files and folders where inserted. ​

Here is an excerpt of the log:


Apr 26 16:50:16 ip-XXX-XXX-XXX-XXX sshd[4209]: Connection closed by 42.116.165.189 port 27920 [preauth]

Apr 26 16:50:23 ip-XXX-XXX-XXX-XXX sshd[4212]: Invalid user admin1 from 42.116.165.189

Apr 26 16:50:23 ip-XXX-XXX-XXX-XXX sshd[4212]: input_userauth_request: invalid user admin1 [preauth]

Apr 26 16:50:24 ip-XXX-XXX-XXX-XXX sshd[4212]: Connection closed by 42.116.165.189 port 46723 [preauth]

Apr 26 16:50:31 ip-XXX-XXX-XXX-XXX sshd[4214]: Invalid user admin1 from 42.116.165.189

Apr 26 16:50:31 ip-XXX-XXX-XXX-XXX sshd[4214]: input_userauth_request: invalid user admin1 [preauth]

Apr 26 16:50:31 ip-XXX-XXX-XXX-XXX sshd[4214]: Connection closed by 42.116.165.189 port 11874 [preauth]

Apr 26 16:50:38 ip-XXX-XXX-XXX-XXX sshd[4216]: Invalid user admin1 from 42.116.165.189

Apr 26 16:50:38 ip-XXX-XXX-XXX-XXX sshd[4216]: input_userauth_request: invalid user admin1 [preauth]

Apr 26 16:50:38 ip-XXX-XXX-XXX-XXX sshd[4216]: Connection closed by 42.116.165.189 port 51476 [preauth]

Apr 26 16:50:46 ip-XXX-XXX-XXX-XXX sshd[4219]: Connection closed by 42.116.165.189 port 12468 [preauth]

Apr 26 16:50:53 ip-XXX-XXX-XXX-XXX sshd[4221]: Connection closed by 42.116.165.189 port 4515 [preauth]

Apr 26 16:51:00 ip-XXX-XXX-XXX-XXX sshd[4223]: Connection closed by 42.116.165.189 port 16832 [preauth]

Apr 26 16:51:07 ip-XXX-XXX-XXX-XXX sshd[4227]: Connection closed by 42.116.165.189 port 29194 [preauth]

Apr 26 16:51:14 ip-XXX-XXX-XXX-XXX sshd[4229]: Connection closed by 42.116.165.189 port 23483 [preauth]

Apr 26 16:51:22 ip-XXX-XXX-XXX-XXX sshd[4234]: Connection closed by 42.116.165.189 port 58145 [preauth]

Apr 26 16:51:29 ip-XXX-XXX-XXX-XXX sshd[4236]: Connection closed by 42.116.165.189 port 58813 [preauth]

Apr 26 16:51:34 ip-XXX-XXX-XXX-XXX sshd[4238]: Invalid user minecraft from 111.230.240.111

Apr 26 16:51:34 ip-XXX-XXX-XXX-XXX sshd[4238]: input_userauth_request: invalid user minecraft [preauth]

Apr 26 16:51:34 ip-XXX-XXX-XXX-XXX sshd[4238]: Received disconnect from 111.230.240.111 port 37040:11: Bye Bye [preauth]

Apr 26 16:51:34 ip-XXX-XXX-XXX-XXX sshd[4238]: Disconnected from 111.230.240.111 port 37040 [preauth]

Apr 26 16:51:37 ip-XXX-XXX-XXX-XXX sshd[4240]: Connection closed by 42.116.165.189 port 48274 [preauth]

Apr 26 16:51:44 ip-XXX-XXX-XXX-XXX sshd[4243]: Connection closed by 42.116.165.189 port 61791 [preauth]

Apr 26 16:51:51 ip-XXX-XXX-XXX-XXX sshd[4250]: Connection closed by 42.116.165.189 port 61440 [preauth]

Apr 26 16:51:59 ip-XXX-XXX-XXX-XXX sshd[4254]: Connection closed by 42.116.165.189 port 30109 [preauth]

Apr 26 16:52:06 ip-XXX-XXX-XXX-XXX sshd[4256]: Connection closed by 42.116.165.189 port 10518 [preauth]

Apr 26 16:52:13 ip-XXX-XXX-XXX-XXX sshd[4262]: Connection closed by 42.116.165.189 port 40676 [preauth]

Apr 26 16:52:20 ip-XXX-XXX-XXX-XXX sshd[4264]: Connection closed by 42.116.165.189 port 9543 [preauth]

Apr 26 16:52:28 ip-XXX-XXX-XXX-XXX sshd[4266]: Connection closed by 42.116.165.189 port 57895 [preauth]

Apr 26 16:52:35 ip-XXX-XXX-XXX-XXX sshd[4270]: Connection closed by 42.116.165.189 port 20229 [preauth]

Apr 26 16:52:42 ip-XXX-XXX-XXX-XXX sshd[4273]: Connection closed by 42.116.165.189 port 61376 [preauth]

Apr 26 16:52:50 ip-XXX-XXX-XXX-XXX sshd[4277]: Connection closed by 42.116.165.189 port 4581 [preauth]

Apr 26 16:52:57 ip-XXX-XXX-XXX-XXX sshd[4279]: Connection closed by 42.116.165.189 port 46025 [preauth]

Apr 26 16:53:04 ip-XXX-XXX-XXX-XXX sshd[4281]: Connection closed by 42.116.165.189 port 50525 [preauth]

Apr 26 16:53:09 ip-XXX-XXX-XXX-XXX sshd[4283]: Received disconnect from 221.181.185.223 port 15103:11: [preauth]

Apr 26 16:53:09 ip-XXX-XXX-XXX-XXX sshd[4283]: Disconnected from 221.181.185.223 port 15103 [preauth]

Apr 26 16:53:11 ip-XXX-XXX-XXX-XXX sshd[4285]: Invalid user ubnt from 42.116.165.189

Apr 26 16:53:11 ip-XXX-XXX-XXX-XXX sshd[4285]: input_userauth_request: invalid user ubnt [preauth]

Apr 26 16:53:12 ip-XXX-XXX-XXX-XXX sshd[4285]: Connection closed by 42.116.165.189 port 7945 [preauth]

Apr 26 16:53:19 ip-XXX-XXX-XXX-XXX sshd[4287]: Invalid user ubnt from 42.116.165.189

Apr 26 16:53:19 ip-XXX-XXX-XXX-XXX sshd[4287]: input_userauth_request: invalid user ubnt [preauth]

Apr 26 16:53:19 ip-XXX-XXX-XXX-XXX sshd[4287]: Connection closed by 42.116.165.189 port 20795 [preauth]

Apr 26 16:53:26 ip-XXX-XXX-XXX-XXX sshd[4289]: Invalid user ubnt from 42.116.165.189

Apr 26 16:53:26 ip-XXX-XXX-XXX-XXX sshd[4289]: input_userauth_request: invalid user ubnt [preauth]

Does it mean that someone with ip 42.116.165.189 tried to gain access and was not able to do so? Is it possible to find who was behind it? I guess the intruder will have used a series of VPNs to hide his identity.

Also how do 111.230.240.111 and 221.181.185.223 come into play? What does [preauth] mean?

thank you so much for your detailed and very helpful answer

My pleasure!

Does it mean that someone with ip 42.116.165.189 tried to gain access and was not able to do so? Is it possible to find who was behind it? I guess the intruder will have used a series of VPNs to hide his identity.

All the entries in your log make reference to sshd, which is the system service that handles SSH connections. That effectively means that all lines you shared are related to events affecting SSH in your instance. More precisely, if you do not recognize 42.116.165.189, its behaviour is clearly suspicious as it tries to establish a SSH connection to the instance using different common usernames (e.g. admin1, minecraft, ubnt, …).

Regarding finding out more information about it, I am not an expert in the matter but I’d say that it is not that trivial as you pointed out. Additionally, the effort does not really pay off as even if you knew it there is little to do. Hence, the most effective countermeasure to these types of bots/attacks is to simply restrict the incoming traffic to the SSH port to a single/range of known IPs. You can do that using inbound firewall rules, and we have a tutorial that covers it in case you find it handy.

https://docs.bitnami.com/aws/faq/administration/use-firewall/

Also how do 111.230.240.111 and 221.181.185.223 come into play? What does [preauth] mean?

I am not completely sure, but it seems they also requested to connect to your instance using SSH and did not provide any username. After a brief period of time, they disconnected. PreAuth means that the event occurred prior to the moment the other endpoint provided details for its authentication.

Best regards,
Jose Antonio Carmona


Was my answer helpful? Click on :heart:

1 Like

Thank @jcarmona again for your answer! This helps someone who is new to these kinds of problems a lot!!
I deleted all the malicious content and unfortunately it was reinstalled some days later. I believe this was not someone’s manual work but there must be some script hidden in our file system or database which recreates the malicious content as soon as it is removed. But I am wondering, Is it actually technically possible to execute a script over an entry within our database or is this something I can exclude as a possible source of the problem? If so I would only concentrate on our file system…

Also do you know about any place in the web where such matters are discussed or where experts in the matter share their knowledge (e.g. something like stackoverflow for hacked software), strategies on how to find the source of it all?

One more question: Is it possible for a hacker to gain access to the entire file system (up until root directory “/”) once you have access as admin to WP backend? Or can you only access everything on the level of “htdocs” (i.e. where wp-content, wp-config.php etc live) and below?

Hi again!

Of course, you could have a script that runs periodically and insert values in the database. Having said that, it is fair if you want to proceed with the investigation, but if you have a backup of your data you don’t need to spend so much time on this if this is not your main priority :slightly_smiling_face:

Also do you know about any place in the web where such matters are discussed or where experts in the matter share their knowledge (e.g. something like stackoverflow for hacked software), strategies on how to find the source of it all?

Unfortunately, I haven’t used those sites actively myself to have a strong opinion and hence I can’t really recommend any.

Is it possible for a hacker to gain access to the entire file system (up until root directory “/”) once you have access as admin to WP backend? Or can you only access everything on the level of “htdocs” (i.e. where wp-content, wp-config.php etc live) and below?

It just depends on what the hacker is able to achieve. Theoretically, the biggest threat would be to have complete access to a machine and impersonate the administrator. In that scenario, the intruder would have full access to the machine just as you do. Nevertheless, there are some countermeasures to prevent that. For example, Bitnami configures its stacks in order for services to run on different Linux users and groups. Impersonating the user/group daemon (the one that Apache uses) would result on restrict access to the files Apache is able to manage (htdocs for example), but not the /opt/bitnami/mysql for instance.

Best regards,
Jose Antonio Carmona


Was my answer helpful? Click on :heart:

Thank you again :slight_smile:

Of course, you could have a script that runs periodically and insert values in the database. Having said that, it is fair if you want to proceed with the investigation, but if you have a backup of your data you don’t need to spend so much time on this if this is not your main priority :slightly_smiling_face:

The Problem is the hack took place already over a year ago and I just noticed it 3 weeks ago… so I unfortunately do not have a backup which goes back that long. My WP-Updraft Plugin only reaches 30 days back…

Is there a general log file within the bitnami installation which logs every action (not only login attempts like the auth.log, but also when a user generates a new post or the like)?

The Problem is the hack took place already over a year ago and I just noticed it 3 weeks ago… so I unfortunately do not have a backup which goes back that long. My WP-Updraft Plugin only reaches 30 days back…

Oh, that is sad to hear :frowning: . In any case, I encourage you to try and perform a backup of the existing status following the guide:
https://docs.bitnami.com/google/apps/wordpress/administration/backup-restore-jetpack/

While maintaining the old instance, try to create a new one and restore the generated backup and see the results. Following this approach, at least you reduce the scope of analysis to the imported data (which comprises the DB and the Wordpress directory).

Is there a general log file within the bitnami installation which logs every action (not only login attempts like the auth.log, but also when a user generates a new post or the like)?

I am afraid there is no specific log that traces all the actions performed in the system. Think that this will generate a gigantic file in terms of size, which in more cases is not convenient with respect to the information it would provide. Nevertheless, you can still inspect some available logfiles to see if you spot something out of place:

Apache Access and Error logfiles:
/opt/bitnami/apache2/logs/

PHP-FPM logfiles:
/opt/bitnami/php/var/log/php-fpm.log or /opt/bitnami/php/logs/php-fpm.log

MySQL logfiles:
/opt/bitnami/mysql/data/mysqld.log or /opt/bitnami/mariadb/logs/mysqld.log

Best regards,
Jose Antonio Carmona


Was my answer helpful? Click on :heart:

I faced hacking attempt too on https://MYSITE.com, I used wordfence and now its gone.

Hi @aarianagrande30

Glad you solved your issue! We usually recommend the use of WordFence to Secure your WordPress installations. In fact, we have a guide that covers the set up process :slightly_smiling_face::

https://docs.bitnami.com/google/apps/wordpress/troubleshooting/enforce-security/

Best regards,
Jose Antonio Carmona


Was my answer helpful? Click on :heart:

I was also finally able to solve the problem by running a series of scans through various tools, such as Wordfence, Sucuri etc. The malicious content has not come back thank God :slight_smile: Unfortunately after I deleted Wordfence again, my site became very slow, returning lots of 504 errors…

I was also finally able to solve the problem by running a series of scans through various tools, such as Wordfence, Sucuri etc. The malicious content has not come back thank God

I am glad to hear you were able to solve your problem :tada: :slightly_smiling_face:

Unfortunately after I deleted Wordfence again, my site became very slow, returning lots of 504 errors…

Could you please create another ticket for this? We tend to have single-themed threads so that is easy for other users to find solutions to specific problems. One of our team members will be more than happy to provide you with assistance on a new thread :slightly_smiling_face:

Best regards,
Jose Antonio Carmona


Was my answer helpful? Click on :heart: