Error Apache and MongoDB auto-start after reboot

Keywords: MEAN - AWS - Technical issue - Other
bnsupport ID: b6a27156-de03-fa7a-ae68-e233ace49183
Description:
Hello everyone,

After the ec2 reboot, neither apache nor Mongo DB go up. I get the following error:

apache 07:53:39.21 ERROR ==> apache did not start
Apr 01 07:53:39 ip-172-31-64-14 bitnami[28263]: 2021-04-01T07:53:39.219Z - error: Unable to perform start operation Export start for apache failed with exit code 1
Apr 01 07:53:39 ip-172-31-64-14 bitnami[28263]: ## 2021-04-01 07:53:39+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/010_bitnami_agent_extra...
Apr 01 07:53:39 ip-172-31-64-14 bitnami[28263]: ## 2021-04-01 07:53:39+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/020_bitnami_agent...
Apr 01 07:53:39 ip-172-31-64-14 bitnami[28263]: ## 2021-04-01 07:53:39+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/030_update_welcome_file...
Apr 01 07:53:39 ip-172-31-64-14 bitnami[28263]: ## 2021-04-01 07:53:39+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/040_bitnami_credentials_file...
Apr 01 07:53:39 ip-172-31-64-14 bitnami[28263]: ## 2021-04-01 07:53:39+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/050_clean_metadata...
Apr 01 07:53:39 ip-172-31-64-14 systemd[1]: bitnami.service: Control process exited, code=exited, status=1/FAILURE
Apr 01 07:53:39 ip-172-31-64-14 systemd[1]: bitnami.service: Failed with result 'exit-code'.
Apr 01 07:53:39 ip-172-31-64-14 systemd[1]: Failed to start LSB: bitnami init script.

support bundle code: b6a27156-de03-fa7a-ae68-e233ace49183

Thanks in advance,
Simone

Hi @simone.tiso,

Thanks for using Bitnami. Can you check the Apache configuration using the next command?

sudo apachectl -t

Hi @gongomgra,

Syntax OK

Hi @gongomgra,

The following configuration should be ok:

Apache disabled
MongoDB enabled (auto-start after reboot too)

but doesn’t work.

thanks for your support.
Simone

Hello @simone.tiso,

I’m not sure I understood what you are trying, do you want to disable Apache? Also, could you please check the output of the following command:

sudo tail -n 30 mongodb/logs/mongodb.log

Regards,
Francisco de Paz

Hi @fdepaz,

yes, I’d like disable Apache, because I’m using apache microservice in kubernetes.

command output:

2021-04-01T15:32:22.438+0000 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2021-04-01T15:32:22.463+0000 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets…
2021-04-01T15:32:22.463+0000 I NETWORK [listener] removing socket file: /opt/bitnami/mongodb/tmp/mongodb-27017.sock
2021-04-01T15:32:22.463+0000 I - [signalProcessingThread] Stopping further Flow Control ticket acquisitions.
2021-04-01T15:32:22.468+0000 I REPL [signalProcessingThread] shutting down replication subsystems
2021-04-01T15:32:22.472+0000 I REPL [signalProcessingThread] Stopping replication reporter thread
2021-04-01T15:32:22.472+0000 I REPL [signalProcessingThread] Stopping replication fetcher thread
2021-04-01T15:32:22.472+0000 I REPL [signalProcessingThread] Stopping replication applier thread
2021-04-01T15:32:22.472+0000 I REPL [rsBackgroundSync] Stopping replication producer
2021-04-01T15:32:22.677+0000 I REPL [rsSync-0] Finished oplog application
2021-04-01T15:32:22.678+0000 I REPL [signalProcessingThread] Stopping replication storage threads
2021-04-01T15:32:22.678+0000 I ASIO [RS] Killing all outstanding egress activity.
2021-04-01T15:32:22.678+0000 I ASIO [RS] Killing all outstanding egress activity.
2021-04-01T15:32:22.678+0000 I ASIO [Replication] Killing all outstanding egress activity.
2021-04-01T15:32:22.679+0000 I ASIO [ReplicaSetMonitor-TaskExecutor] Killing all outstanding egress activity.
2021-04-01T15:32:22.680+0000 I CONTROL [signalProcessingThread] Shutting down free monitoring
2021-04-01T15:32:22.680+0000 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture
2021-04-01T15:32:22.684+0000 I STORAGE [signalProcessingThread] Deregistering all the collections
2021-04-01T15:32:22.685+0000 I STORAGE [WTOplogJournalThread] Oplog journal thread loop shutting down
2021-04-01T15:32:22.685+0000 I STORAGE [signalProcessingThread] Timestamp monitor shutting down
2021-04-01T15:32:22.687+0000 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
2021-04-01T15:32:22.688+0000 I STORAGE [signalProcessingThread] Shutting down session sweeper thread
2021-04-01T15:32:22.688+0000 I STORAGE [signalProcessingThread] Finished shutting down session sweeper thread
2021-04-01T15:32:22.688+0000 I STORAGE [signalProcessingThread] Shutting down journal flusher thread
2021-04-01T15:32:22.738+0000 I STORAGE [signalProcessingThread] Finished shutting down journal flusher thread
2021-04-01T15:32:22.738+0000 I STORAGE [signalProcessingThread] Shutting down checkpoint thread
2021-04-01T15:32:22.739+0000 I STORAGE [signalProcessingThread] Finished shutting down checkpoint thread
2021-04-01T15:32:22.800+0000 I STORAGE [signalProcessingThread] shutdown: removing fs lock…
2021-04-01T15:32:22.800+0000 I CONTROL [signalProcessingThread] now exiting
2021-04-01T15:32:22.800+0000 I CONTROL [signalProcessingThread] shutting down with code:0

Thanks for your support.
Simone

Hello @simone.tiso,

To disable Apache so it doesn’t start after reboot you can run the following commands:

sudo /opt/bitnami/ctlscript.sh stop apache
sudo mv /opt/bitnami/apache2/scripts/ctl.sh /opt/bitnami/apache2/scripts/ctl.sh.disabled

Regards,
Francisco de Paz

Hello @fdepaz,

I just have the following path: /opt/bitnami/scripts/apache, containing the following files:

-rwxr-xr-x 1 root root 569 May 26 2020 stop.sh
-rwxr-xr-x 1 root root 389 May 26 2020 status.sh
-rwxr-xr-x 1 root root 613 May 26 2020 start.sh
-rwxr-xr-x 1 root root 381 May 26 2020 setup.sh
-rwxr-xr-x 1 root root 399 May 26 2020 run.sh
-rwxr-xr-x 1 root root 351 May 26 2020 restart.sh
-rw-r–r-- 1 root root 443 May 26 2020 resize.sh
-rwxr-xr-x 1 root root 391 May 26 2020 reload.sh
-rwxr-xr-x 1 root root 4200 May 26 2020 postunpack.sh
drwxr-xr-x 2 root root 4096 May 26 2020 bitnami-templates
drwxr-xr-x 6 root root 4096 May 26 2020 …
drwxr-xr-x 3 root root 4096 Feb 25 14:15 .

What should I do? What about MongoDB?

many thanks,
Simone

Hello @simone.tiso,

My bad, I thought you were using the old filesystem. Please try changing the following lines on /etc/gonit/gonitrc after stopping Apache:

$ sudo vim /etc/gonit/gonitrc

...
# include /etc/monit/conf.d/*.conf
include /etc/monit/conf.d/mongo.conf

After that, please restart your instance. You should no longer have Apache initiated or appearing in ctlscript.sh status.

Regards,
Francisco de Paz

Hello @fdepaz,

I did. Apache is not running. --> OK
MongoDB is not running --> IT SHOULD

if I run
ctlscript.sh status
I get:

Cannot find any running daemon to contact. If it is running, make sure you are pointing to the right pid file (/var/run/gonit.pid)

Mongo logs:

2021-04-08T14:52:48.323+0000 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2021-04-08T14:52:48.337+0000 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets…
2021-04-08T14:52:48.341+0000 I NETWORK [listener] removing socket file: /opt/bitnami/mongodb/tmp/mongodb-27017.sock
2021-04-08T14:52:48.341+0000 I - [signalProcessingThread] Stopping further Flow Control ticket acquisitions.
2021-04-08T14:52:48.342+0000 I REPL [signalProcessingThread] shutting down replication subsystems
2021-04-08T14:52:48.342+0000 I REPL [signalProcessingThread] Stopping replication reporter thread
2021-04-08T14:52:48.343+0000 I REPL [signalProcessingThread] Stopping replication fetcher thread
2021-04-08T14:52:48.343+0000 I REPL [signalProcessingThread] Stopping replication applier thread
2021-04-08T14:52:48.343+0000 I REPL [rsBackgroundSync] Stopping replication producer
2021-04-08T14:52:48.708+0000 I REPL [rsSync-0] Finished oplog application
2021-04-08T14:52:48.708+0000 I REPL [signalProcessingThread] Stopping replication storage threads
2021-04-08T14:52:48.708+0000 I ASIO [RS] Killing all outstanding egress activity.
2021-04-08T14:52:48.709+0000 I ASIO [RS] Killing all outstanding egress activity.
2021-04-08T14:52:48.710+0000 I ASIO [Replication] Killing all outstanding egress activity.
2021-04-08T14:52:48.710+0000 I ASIO [ReplicaSetMonitor-TaskExecutor] Killing all outstanding egress activity.
2021-04-08T14:52:48.712+0000 I CONTROL [signalProcessingThread] Shutting down free monitoring
2021-04-08T14:52:48.712+0000 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture
2021-04-08T14:52:48.716+0000 I STORAGE [signalProcessingThread] Deregistering all the collections
2021-04-08T14:52:48.717+0000 I STORAGE [WTOplogJournalThread] Oplog journal thread loop shutting down
2021-04-08T14:52:48.718+0000 I STORAGE [signalProcessingThread] Timestamp monitor shutting down
2021-04-08T14:52:48.718+0000 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
2021-04-08T14:52:48.718+0000 I STORAGE [signalProcessingThread] Shutting down session sweeper thread
2021-04-08T14:52:48.718+0000 I STORAGE [signalProcessingThread] Finished shutting down session sweeper thread
2021-04-08T14:52:48.718+0000 I STORAGE [signalProcessingThread] Shutting down journal flusher thread
2021-04-08T14:52:48.816+0000 I STORAGE [signalProcessingThread] Finished shutting down journal flusher thread
2021-04-08T14:52:48.816+0000 I STORAGE [signalProcessingThread] Shutting down checkpoint thread
2021-04-08T14:52:48.816+0000 I STORAGE [signalProcessingThread] Finished shutting down checkpoint thread
2021-04-08T14:52:48.851+0000 I STORAGE [signalProcessingThread] shutdown: removing fs lock…
2021-04-08T14:52:48.851+0000 I CONTROL [signalProcessingThread] now exiting
2021-04-08T14:52:48.851+0000 I CONTROL [signalProcessingThread] shutting down with code:0

Simone

Hi @simone.tiso,

In that case let’s revert the changes on vim /etc/gonit/gonitrc, restart your instance and run the following commands:

sudo /opt/bitnami/ctlscript.sh stop apache
sudo mv /etc/monit/conf.d/apache.conf /etc/monit/conf.d/apache.conf.back

After that, restart again your machine. I have tested this approach and now only mongodb is running:

sudo /opt/bitnami/ctlscript.sh status
mongodb already running

Regards,
Francisco de Paz

hi @fdepaz,

I don’t know why, but, when I run the following command:

sudo /opt/bitnami/ctlscript.sh status

I get:

Cannot find any running daemon to contact. If it is running, make sure you are pointing to the right pid file (/var/run/gonit.pid)

btw, mongo is non running.

thanks for your support,
Simone

Hello @simone.tiso,

Did you revert your gonitrc file to the initial state:

set httpd port 2812 and
    use address localhost
    allow localhost

include /etc/monit/conf.d/*.conf

Could you share the output of the following output:

sudo monit status

Instead of restarting the whole instance, you can run sudo gonit reload after you perform the changes from my previous message.

Regards,
Francisco de Paz

Hi @fdepaz,

yes, I reverted as you told me.

I run

sudo monit status

result:

Cannot find any running daemon to contact. If it is running, make sure you are pointing to the right pid file (/var/run/gonit.pid)

thanks for your support,
Simone

Hi @simone.tiso,

Any issues with the daemon initialization should be solved with sudo gonit reload or sudo gonit:

$ sudo gonit
daemon with PID 1558 awakened

If you get the following error

Socket file '/var/run/gonit.sock' is not writable

Remove that file:

sudo rm /var/run/gonit.sock

And try again:

sudo gonit

And check the status:

sudo /opt/bitnami/ctlscript.sh status

Does the error persists if you leave /etc/gonit/gonitrc to include both Apache and MongoDB with include /etc/monit/conf.d/*.conf?

I have also seen an older thread where you encounter the same issue, could you try following the same steps and revoking the certificates?

Regards,
Francisco de Paz

Hi @fdepaz,

I followed your advise:

bitnami@:~$ sudo monit status
Cannot find any running daemon to contact. If it is running, make sure you are pointing to the right pid file (/var/run/gonit.pid)
bitnami@:~$ sudo gonit
Starting gonit daemon
bitnami@:~$ sudo /opt/bitnami/ctlscript.sh status
mongodb not running

It seems neither gonit nor MongoDB go up after reboot.

If this can be useful, I write down the bitnami.service logs:

Apr 12 15:08:54 ip-172-31-64-14 systemd[1]: Starting LSB: bitnami init script…
Apr 12 15:08:54 ip-172-31-64-14 bitnami[561]: ## 2021-04-12 15:08:54+02:00 ## INFO ## Running /opt/bitnami/var/init/pre-start/010_resize_fs…
Apr 12 15:08:54 ip-172-31-64-14 bitnami[561]: resize2fs 1.44.5 (15-Dec-2018)
Apr 12 15:08:54 ip-172-31-64-14 bitnami[561]: resize2fs: Device or resource busy while trying to open /dev/xvda
Apr 12 15:08:54 ip-172-31-64-14 bitnami[561]: Couldn’t find valid filesystem superblock.
Apr 12 15:08:54 ip-172-31-64-14 bitnami[561]: ## 2021-04-12 15:08:54+02:00 ## INFO ## Running /opt/bitnami/var/init/pre-start/020_hostname…
Apr 12 15:08:54 ip-172-31-64-14 bitnami[561]: ## 2021-04-12 15:08:54+02:00 ## INFO ## Running /opt/bitnami/var/init/pre-start/030_swap_file…
Apr 12 15:08:54 ip-172-31-64-14 bitnami[561]: ## 2021-04-12 15:08:54+02:00 ## INFO ## Running /opt/bitnami/var/init/pre-start/040_check_if_demo_machine…
Apr 12 15:08:54 ip-172-31-64-14 bitnami[561]: ## 2021-04-12 15:08:54+02:00 ## INFO ## Running /opt/bitnami/var/init/pre-start/050_change_boot_log_permissions…
Apr 12 15:08:54 ip-172-31-64-14 bitnami[561]: ## 2021-04-12 15:08:54+02:00 ## INFO ## Running /opt/bitnami/var/init/pre-start/060_get_default_passwords…
Apr 12 15:08:54 ip-172-31-64-14 bitnami[561]: ## 2021-04-12 15:08:54+02:00 ## INFO ## Starting services…
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: 2021-04-12T13:08:57.108Z - info: Saving configuration info to disk
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: 2021-04-12T13:08:57.117Z - debug: Initialized options { cloudName: ‘aws’,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: instanceTier: ‘main’,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: instanceData: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: peerAddress: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: peerPort: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: peerUsername: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: peerConnectionOptions: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: publicEndpoint: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: extraServices: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: appUsername: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: appPassword: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: appDatabase: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: appRepository: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: clusterName: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: clusterQuorum: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: persistentNode: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: booleanInput: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: configOverrides: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: snitch: undefined,
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: peerNodes: { prefix: ‘’, startIndex: 0, index: 0, count: 0 },
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: storageAccountName: undefined }
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: 2021-04-12T13:08:57.665Z - info: Data disk not present
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: 2021-04-12T13:08:57.669Z - info: Performing service start operation for apache
Apr 12 15:08:57 ip-172-31-64-14 bitnami[561]: 2021-04-12T13:08:57.675Z - verbose: Running export start for apache
Apr 12 15:09:57 ip-172-31-64-14 bitnami[561]: apache 15:09:57.82 ERROR ==> apache did not start
Apr 12 15:09:57 ip-172-31-64-14 bitnami[561]: 2021-04-12T13:09:57.821Z - error: Unable to perform start operation Export start for apache failed with exit code 1
Apr 12 15:09:57 ip-172-31-64-14 bitnami[561]: ## 2021-04-12 15:09:57+02:00 ## INFO ## Running /opt/bitnami/var/init/post-start/010_bitnami_agent_extra…
Apr 12 15:09:57 ip-172-31-64-14 bitnami[561]: ## 2021-04-12 15:09:57+02:00 ## INFO ## Running /opt/bitnami/var/init/post-start/020_bitnami_agent…
Apr 12 15:09:57 ip-172-31-64-14 bitnami[561]: ## 2021-04-12 15:09:57+02:00 ## INFO ## Running /opt/bitnami/var/init/post-start/030_update_welcome_file…
Apr 12 15:09:57 ip-172-31-64-14 bitnami[561]: ## 2021-04-12 15:09:57+02:00 ## INFO ## Running /opt/bitnami/var/init/post-start/040_bitnami_credentials_file…
Apr 12 15:09:57 ip-172-31-64-14 bitnami[561]: ## 2021-04-12 15:09:57+02:00 ## INFO ## Running /opt/bitnami/var/init/post-start/050_clean_metadata…
Apr 12 15:09:57 ip-172-31-64-14 systemd[1]: bitnami.service: Control process exited, code=exited, status=1/FAILURE
Apr 12 15:09:57 ip-172-31-64-14 systemd[1]: bitnami.service: Failed with result ‘exit-code’.
Apr 12 15:09:57 ip-172-31-64-14 systemd[1]: Failed to start LSB: bitnami init script.

Does the error persists if you leave /etc/gonit/gonitrc to include both Apache and MongoDB with include /etc/monit/conf.d/*.conf?

Yes

I have also seen an older thread where you encounter the same issue, could you try following the same steps and revoking the certificates?

I have already revoked all certificates.

thanks for your support,
Simone

Hello @simone.tiso,

Let me try to reproduce the issue and come back with my findings, thanks!

Regards,
Francisco de Paz

Hi @fdepaz,

ok, thanks. I noticed the service gonit doesn’t go up automatically on EC2 start up. should it do?

On startup, if I run:

sudo gonit

everything works, MongoDB too.

Simone.

Hi @simone.tiso,

Great! I’m glad everything is working as expected now! If you encounter any more issues on your instance do not doubt to open a new thread so we can keep helping you.

Regards,
Francisco de paz

Hi @fdepaz,

yes, thanks, but I’m still getting issues with this AMI.

service gonit doesn’t go up automatically on EC2 start up

I need to run the command after every EC2 reboot. This is a problem for us.

thanks in advance.
Simone