Bitnami WordPress on AWS EC2 just stopped working

wbpaley 2021-02-22 21:29:20 UTC #1

Keywords: WordPress - AWS - Technical issue - Other
Description:
The site http://didi.co just stopped--I have not edited or changed anything in a week or more. I tried restarting the instance several times and no luck. (It stopped on the 16th but restarting the instance fixed it)

This article's response mentions a Bitnami support tool that might help--can I use it or can someone use it and look at my site/instance? Thanks in advance!

https://community.bitnami.com/t/my-wordpress-aws-instance-just-stopped-working/84152

wbpaley 2021-02-22 21:32:03 UTC #2

By the way, when I created the topic it suggested I run a tool and provided an id BUT THERE WAS NO LINK I COULD SEE ANYWHERE TO RUN THE TOOL. Either I'm blind or Chrome on a PC didn't display it somehow--I've had similar issues with it before...

This was the text on the Bitnami topic-posting page that I couldn't comply with, though I wanted to:

To ensure we can quickly provide you with a resolution to your question, please run this tool in the affected environment. After running the tool, enter the resulting code in the box below. You can find more information here.

123456789-abcd-1234-abcd-123456789abc

jota 2021-02-23 11:54:08 UTC #3

Hi @wbpaley,

You need to run the Bitnami Support tool in the instance where WordPress is running. That tool will gather relevant information for us to analyze your configuration and logs.

How to Run the Bitnami Support Tool in a cloud image or virtual machine

You need to paste here the ID you obtain when running the tool.

If you can't run the tool, please run these other commands for us to get more information about your instance.

free -m
df -H
sudo /opt/bitnami/ctlscript.sh status

Thanks

wbpaley 2021-03-01 21:17:33 UTC #4

I did this a few days ago and responded to the above directly in the email that this system sent me. (It said in the email I could do that.)

No reply. Am I being too impatient? (But pardon: my business is seriously impacted by this!)

Thanks,
Brad

I repeat my email body, just in case, here:

Jota,

Please respond or ask someone else to do so. My business depends on this.

I believe the initial problem was that I ran into the limit of my 10GB storage.

So I increased it to 50GB and stopped and started the instance and it is now running on AWS at ip 54.157.53.221.

But Wordpress is still not running.

And I can't get into mysql.
(The password that worked for Wordpress does not work. I thought they were set to the same password?)
(If I can somehow get into mysql, a local Wordpress expert says he can move my site to a new instance)

And I can't change the root mysql password.
I tried following these directions several times:
https://docs.bitnami.com/aws/infrastructure/lamp/administration/change-reset-password
I always get:
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

But I was able to run the Bitnami Support Tool, here's the ID:
8cb5b52a-ea09-d117-e796-72754cc4f4e6

Please, I am very concerned. What can I do?

I will pay if I have to get higher-priority service--is that necessary? And how do I do that?

Thank You,
Brad

Then, just after that:

Also, if it is helpful, I ran the commands you told me again (result below).

And I attach the log from the AWS instance console that should have the password information in it.
(It doesn't seem to on this instance, it does on my other, fresh, AWSEC2/Bitnami/Wordpress instance.)

bitnami@ip-172-31-94-206:~$ free -m
total used free shared buff/cache available
Mem: 990 364 320 46 305 386
Swap: 634 147 487
bitnami@ip-172-31-94-206:~$ df -H
Filesystem Size Used Avail Use% Mounted on
udev 512M 0 512M 0% /dev
tmpfs 104M 3.4M 101M 4% /run
/dev/xvda1 52G 9.5G 43G 19% /
tmpfs 520M 0 520M 0% /dev/shm
tmpfs 5.3M 0 5.3M 0% /run/lock
tmpfs 520M 0 520M 0% /sys/fs/cgroup
/dev/loop0 59M 59M 0 100% /snap/core18/1988
/dev/loop1 59M 59M 0 100% /snap/core18/1944
/dev/loop2 103M 103M 0 100% /snap/core/10583
/dev/loop3 104M 104M 0 100% /snap/core/10823
/dev/loop4 35M 35M 0 100% /snap/amazon-ssm-agent/3552
/dev/loop5 34M 34M 0 100% /snap/amazon-ssm-agent/2996
tmpfs 104M 0 104M 0% /run/user/1000
bitnami@ip-172-31-94-206:~$ sudo /opt/bitnami/ctlscript.sh status
php-fpm already running
apache already running
mysql already running
bitnami@ip-172-31-94-206:~$

wbpaley 2021-03-01 21:21:07 UTC #5

Also, trying to get back on line!, I tried another way to reset my root password, following the different instructions here:
https://phoenixnap.com/kb/how-to-reset-mysql-root-password-windows-linux

This is what I see after killing the mysqld processes:

bitnami@ip-172-31-94-206:~$ sudo /opt/bitnami/ctlscript.sh status
php-fpm already running
apache already running
mysql not running
bitnami@ip-172-31-94-206:~$ mysqld --init-file=/home/me/mysql-init &
[1] 24281
bitnami@ip-172-31-94-206:~$ 2021-03-01T20:57:40.773260Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2021-03-01T20:57:40.773381Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld.bin (mysqld 8.0.18) starting as process 24281
2021-03-01T20:57:40.776666Z 0 [Warning] [MY-010091] [Server] Can't create test file /opt/bitnami/mysql/data/mysqld_tmp_file_case_insensitive_test.lower-test
2021-03-01T20:57:40.776679Z 0 [Warning] [MY-010159] [Server] Setting lower_case_table_names=2 because file system for /opt/bitnami/mysql/data/ is case insensitive
2021-03-01T20:57:40.776719Z 0 [ERROR] [MY-013276] [Server] Failed to set datadir to '/opt/bitnami/mysql/data/' (OS errno: 13 - Permission denied)
2021-03-01T20:57:40.777129Z 0 [ERROR] [MY-010119] [Server] Aborting
2021-03-01T20:57:40.777253Z 0 [System] [MY-010910] [Server] /opt/bitnami/mysql/bin/mysqld.bin: Shutdown complete (mysqld 8.0.18) MySQL Community Server - GPL.
mysqld --init-file=/home/me/
[1]+ Exit 1 mysqld --init-file=/home/me/mysql-init
bitnami@ip-172-31-94-206:~$

jota 2021-03-02 07:58:55 UTC #6

Hi @wbpaley,

wbpaley:

bitnami@ip-172-31-94-206:~$ free -m total used free shared buff/cache availableMem: 990 364 320 46 305 386Swap: 634 147 487bitnami@ip-172-31-94-206:~$ df -HFilesystem Size Used Avail Use% Mounted onudev 512M 0 512M 0% /devtmpfs 104M 3.4M 101M 4% /run/dev/xvda1 52G 9.5G 43G 19% /tmpfs 520M 0 520M 0% /dev/shmtmpfs 5.3M 0 5.3M 0% /run/locktmpfs 520M 0 520M 0% /sys/fs/cgroup/dev/loop0 59M 59M 0 100% /snap/core18/1988/dev/loop1 59M 59M 0 100% /snap/core18/1944/dev/loop2 103M 103M 0 100% /snap/core/10583/dev/loop3 104M 104M 0 100% /snap/core/10823/dev/loop4 35M 35M 0 100% /snap/amazon-ssm-agent/3552/dev/loop5 34M 34M 0 100% /snap/amazon-ssm-agent/2996tmpfs 104M 0 104M 0% /run/user/1000bitnami@ip-172-31-94-206:~$ sudo /opt/bitnami/ctlscript.sh statusphp-fpm already runningapache already runningmysql already runningbitnami@ip-172-31-94-206:~$

It seems all the services are running now and you have free space in the disk. However, I can see you are using a 1GB of RAM instance and it's already using SWAP so that can be a potential issue in the future. I suggest you increase the memory of the machine if you continue running into performance issues in the future.

Why are you trying to reset the MySQL's root password? Let's try to access the database using the credentials that are configured in the WordPress' conf file. Can you open the /opt/bitnami/apps/wordpress/htdocs/wp-config.php file and get the password to use to connect to the database?

mysql -u bn_wordpress -p bitnami_wordpress

Note: you will need to enter the password now

Was the connection successful?

Apart from that, I just accessed your site and I can see the WordPress app is running properly so there shouldn't be any database issue. What was the problem you ran into?

Thanks

wbpaley 2021-03-04 01:36:31 UTC #7

Hi Jota--thanks!

First, I'd have responded instantly if I knew you had looked into this yesterday--this is critical to me as you know. I expected an email notification (I don't live on the Bitnami support site!) like I got for your first email. Do I need to configure something to get an email notification for each of your responses? Or can you just email me directly? (I do live on email... ; )

I have no idea how that 54.157.53.221 ip address got mapped to point to my new, one-page temporary "Please pardon our pause; back up soon." Wordpress instance that you saw and sent me an image of. I created that temp page on a new AWS instance (3.82.176.144) just so clients would see something instead of a "This site can’t be reached" error message. I only created that temp page once, on 3.82.176.144, and did not copy it to the original instance that was at 54.157.53.221 because I was terrified I'd lose all the work on that original instance. A friend says sometimes services point old ip addresses at new instances, so I suppose something like that happened.

Regardless, here's the situation now: I had an EC2 snapshot I made of the original instance after it stopped working--but I believe it has all of my work on it. And I've been able to launch a new instance from an image made from that snapshot, it's at ip address 52.87.242.150. I increased its disk space to 50GB and tried to reboot the the instance, hoping that would just restart everything, but no luck. However I can ssh into it, and so I ran /opt/bitnami/bnsupport-tool and it gave me this code:

0b30fb60-eab3-3609-7780-5a9a4e5b4503

Now I have two options: get the new instance (52.87.242.150) I created with the snapshot of the broken Wordpress running again--or migrate the site to another instance. I prefer to get the snapshot-restored one running if I can.

1) Can you help me get it running?

If not, I have someone who can migrate from it if I can log in to mysql but I can't--no passwords that I have work. That was why I was trying to reset the mysql password.

So if you can't help me do 1) above can you help me do 2) below:

2) Reset the mysql password.

(I have not tried to follow any Web directions to do this on the newly-restores-from-snapshot instance 52.87.242.150 because I failed for hours trying to do it before--and have found differing instructions on the Web--and don't want to mess up this instance...)

Pardon all the detail--hope I covered what you need to help me!

Thanks!
Brad

wbpaley 2021-03-04 01:52:42 UTC #8

Jota:

Pardon: I missed your reference to the wp-config.php file.

I used that userid and password and got into mysql!

I have send them to the person who can (at a cost...) migrate me; I hope he doesn't need to log in as root.

So I will do my plan B: migrate to another instance.

Unless you can save me significant migration money by helping me get the restored-snapshot 52.87.242.150 instance running? That's the one I ran /opt/bitnami/bnsupport-tool as I said above and got code:

0b30fb60-eab3-3609-7780-5a9a4e5b4503

I don't mind running plan A with you and plan B with the wordpress guy in parallel; just gotta get one (or both!) of these running.

Thanks!
Brad

jota 2021-03-04 11:32:23 UTC #9

Hi @wbpaley,

You should be receiving email notifications with every reply we post. If you are not receiving it, we will review the configuration of the forum to ensure they are sent.

Ok, the 54.157.53.221 instance is redirecting to the domain and the domain is pointing to the temp instance you launched

$ curl -LI 54.157.53.221
HTTP/1.1 301 Moved Permanently
Date: Thu, 04 Mar 2021 11:23:58 GMT
Server: Apache
X-Powered-By: PHP/7.3.14
X-Redirect-By: WordPress
X-Frame-Options: SAMEORIGIN
Vary: Cookie
Location: http://didi.co/
Content-Type: text/html; charset=UTF-8

https://www.whatsmydns.net/#A/didi.co

I can see that you ran into space issues on Feb 23rd

[Tue Feb 23 06:29:40.813468 2021] [log_config:warn] [pid 4560:tid 140322632054528] (28)No space left on device: [client 172.31.43.193:44882] AH00646: Error writing to logs/access_log
...
[Tue Feb 23 10:14:56.359228 2021] [pagespeed:error] [pid 1932:tid 140026536883968] [mod_pagespeed 1.13.35.2-0 @1932] /opt/bitnami/apache2/logs/pagespeed_log/stats_log_global: flushing file 28(No space left on device)                                                        [Tue Feb 23 10:14:56.372953 2021] [proxy:error] [pid 1932:tid 140026536883968] (111)Connection refused: AH02454: FCGI: attempt to connect to Unix domain socket /opt/bitnami/php/var/run/wordpress.sock (wordpress-fpm) failed

but you increased the disk space and that's not a problem now.

-----------------------------------
Check free amount of disk space
-----------------------------------
Running: df -h
In: /opt/bitnami

Output:

Filesystem      Size  Used Avail Use% Mounted on
udev            488M     0  488M   0% /dev
tmpfs           100M  3.3M   96M   4% /run
/dev/xvda1       49G  8.8G   40G  19% /

However, I tried to access that new instance (52.87.242.150) but the firewall configuration seems to be blocking the requests to the ports 80 and 443

-----------------------------------
Check if port 80 is externally accessible:
-----------------------------------
Running: nc -zv -w5 52.87.242.150 80
In: /opt/bitnami
Error: nc: connect to 52.87.242.150 port 80 (tcp) timed out: Operation now in progress
Exitcode: 1
....................................
....................................
-----------------------------------
Check if port 443 is externally accessible:
-----------------------------------
Running: nc -zv -w5 52.87.242.150 443
In: /opt/bitnami
Error: nc: connect to 52.87.242.150 port 443 (tcp) timed out: Operation now in progress
Exitcode: 1

Please ensure ports 22, 80 and 443 are open in the firewall

https://docs.bitnami.com/aws/faq/administration/use-firewall/

Once you do that, you should be able to access your application using that IP address

Did you try to access the database using the WordPress credentials in the wp-config.php file?

mysql -u bn_wordpress -p bitnami_wordpress

wbpaley 2021-03-04 17:17:53 UTC #10

Jota: I AM UP AND RUNNING!

Many thanks.

(Yes, FYI: I didn't get an email notification for this reply either. Happy to act as a test subject if you like.)

Grateful Regards,
Brad

jota 2021-03-05 11:09:23 UTC #11

I'm really glad to hear that.

Regarding the notifications, I'm going to create an internal task and we will take a look at the application to ensure the notifications are sent.

Enjoy

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!