Gonit not initializing / monitoring on startup

lets4done 2020-12-10 06:41:45 UTC #1

Keywords: Drupal - Google Cloud Platform - Technical issue - Other
bnsupport ID: d14541ba-a6ef-80c3-6369-69f59b6a40a6
Description:
1. Created a brand instance of drupal 9.1 image on GCP (Google cloud platform) for this test case.
2. Installed google monitor agent to watch ram usage to make sure not running out of memory. restarted instance. It is not. steady at 40%
3. sudo gonit status command says is monitoring status apache, mariadb, and php-fpm but then to the right has "not monitored"

Am I not understanding what gonit should be doing out of the box? on startup??

Now if I restart or start ie sudo gonit start it changes its status to monitored and lists pid, uptime and other info as expected.

More info:
ctlscript and apache config status seem to be doing similar issues.
I have verified web pages are being served (and not cached) so the site is working, just gonit is not reporting / monitoring it correctly.
The basic LAMP bitnami stack does not duplicate this issue.
If I stop and restart the GCP instance it goes back to same old issue.

Here is a bnsupport #
d14541ba-a6ef-80c3-6369-69f59b6a40a6

Thanks for the use of this image btw.
Look forward to getting this fixed and maybe helping the community out.... (unless I am way off base and then you will update me)

Ctlscript status shows as daemon not running

Ibone 2020-12-10 10:42:32 UTC #2

Hi @lets4done,

Thanks for using Bitnami. Did you follow this guide https://cloud.google.com/monitoring/agent/installation#vms_without_remote_package_access to install Cloud Monitoring agent?

I'm trying to reproduce the problem but I can't, could you tell us what steps are you follow to get that status of gonit?

Could you copy the output of the following commands?

sudo /opt/bitnami/ctlscript.sh  status
sudo gonit status
sudo service stackdriver-agent status

Regards,
Ibone.

lets4done 2020-12-10 18:33:44 UTC #3

IT WORKS NOW! Thank you. (yea was shouting, but not yelling, ie happy, so ok?)
at least for new instances

But now how do I fix it for my existing instances. When ever I stop an instance then restart gonit is doing same thing described earlier.

Pls ignore stackdriver agent portion.... it is just confusing issue. I only included it to show memory was not an issue (at least as far as instance goes)(yes those are the directions I followed for installing stackdriver agent btw)

Could you copy the output of the following commands?

From another previous instance:

$ sudo /opt/bitnami/ctlscript.sh  status
apache not running
mariadb not running
php-fpm not running

$ sudo gonit status

Uptime                         15s
Last Check                     2020-12-10 18:17:27.586941562 +0000 UTC m=+0.255585073
Next Check                     2020-12-10 18:19:27.586941562 +0000 UTC m=+120.255585073
Pid                            1434
Pid File                       /var/run/gonit.pid
Control File                   /etc/gonit/gonitrc
Socket File                    /var/run/gonit.sock
Log File                       /var/log/gonit.log
Process 'apache'
  monitoring status                        Not monitored

Process 'mariadb'
  monitoring status                        Not monitored

Process 'php-fpm'
  monitoring status                        Not monitored

Thanks for your assistance in this.

Ibone 2020-12-11 08:32:44 UTC #4

Hi @lets4done,

I'm glad that works

It's strange in your data of the bnsupport, the services appear running, Could you run the following command and if appears an error added in the topic?

sudo /opt/bitnami/ctlscript.sh start

Regards,
Ibone.

lets4done 2020-12-12 06:28:56 UTC #5

I have a bit of an update below.
For your request

sudo /opt/bitnami/ctlscript.sh start

returns

Starting services..

When I type

sudo /opt/bitnami/ctlscript.sh status

It returns

apache not running
mariadb not running
php-fpm not running

Now if i start it with

sudo gonit start

it doesn't respond with anything but it does start monitoring as expected. But only while the instatance is live. If it is stopped and started or reset, ie from GCP Compute engine instance interface, it goes back to service "not running".

sudo gonit status

It works as expected for now, until instance is cycled.

Uptime 10m3s
Last Check 2020-12-12 04:25:48.336255777 +0000 UTC m=+600.049049352
Next Check 2020-12-12 04:27:48.336255777 +0000 UTC m=+720.049049352
Pid 1234
Pid File /var/run/gonit.pid
Control File /etc/gonit/gonitrc
Socket File /var/run/gonit.sock
Log File /var/log/gonit.log
Process 'apache'
status Running
pid 123
uptime 10m3s
monitoring status monitored

Process 'mariadb'
status Running
pid 12
uptime 10m3s
monitoring status monitored

Process 'php-fpm'
status Running
pid 43
uptime 10m3s
monitoring status monitored
(I changed the PID #...)

So what i have learned so far. Facts

Basic LAMP stack from bitnami does not have this problem at all. Gonit starts and monitors all the time no matter if instances are restarted it restarts properly as I would expect.
For the drupal bitnami image on first boot from a fresh image, gonit works as expected. (wasn't always but it is now)
ctlscript.sh does not seem to start. It will report status, but doesn't start monitoring.
This is happening on all drupal 9 images. existing ones I had previously and newly created ones.

The basic LAMP stack works. Drupal does not. ok so I tried to compare the two. So far have not found differences (Process ids are different etc, ) Looked for file names, directories, file permissions, ownership. Ports and processes seem to be the same (identical) but observably am missing something!
What I didn't do is compare them programmaticly. Might take me a week or 2 to learn how to do just that

Briefly looked through startup script on google for that instance, tried to look in the init for the OS and Bitnami's (/opt/bitnami/var/init & /opt/bitnami/scripts/init) but not sure am looking in the right spot.
Did a grep -a "gonit" and a * if I didn't run out of memory or specified a file might be in.

So I guess I need help. gulp

lets4done 2020-12-12 07:55:59 UTC #6

Just to make it clear, steps to reproduce the issue:

from GCP marketplace search for bitnami and drupal. Only 1 is found, Launch/deploy that
SSH into instance
sudo gonit status
sudo /opt/bitnami/ctlscript.sh status
They both should respond that they are monitoring all 3 services. Ok all is good so far
From GCP VM instances (or Gloud etc) stop and then restart (or reset) the VM.
SSH into instance again
sudo gonit status
sudo /opt/bitnami/ctlscript.sh status

Now this time nothing is being monitored until you do 'sudo gonit start'
No other commands or steps are done.
So far this is reproducible each time for me.
Regards

Ibone 2020-12-15 10:31:05 UTC #7

Hi @lets4done,

You have a small instance with 500MB of memory RAM, probably in at some point it reached its peak of use. This type the instance has a very low performer. We recommend upgrading the instance https://docs.bitnami.com/google/faq/administration/change-server-type/

Regards,
Ibone.

lets4done 2020-12-15 12:34:50 UTC #8

Ok I will do a upgrade to see if that's the issue. In earlier tests ram for the vm instance was steady at 40%. So didn't think there was a problem. But maybe there is another limit??

Ie know about the php memory size setting.
If I upgrade the image will the 'other' limits reflect the change?
What other memory limits should I check?
Thanks

lets4done 2020-12-16 07:06:16 UTC #9

I still have the above questions....

I upgraded and still no workie

I created a new machine:

Machine type: n1-standard-2 (2 vCPUs, 7.5 GB memory)

I know its overkill, but just for testing.

sudo gonit status

it worked!

Stop and restart instance and I think this is where started to fail when stopped and then restarted.
These are the steps I took each time:

sudo gonit start
sudo gonit status
verified running /monitored
stopped from GCP VM
started instance from GCP vm
sudo gonit status

The first time tried php settings as is 256M as stated above. I verified was looking at correct file by

php -i | grep 'Configuration File'

and verified file was being read and reflected new changes by

php -i | grep 'mem'

and looked at memory_limit. I tried setting the php memory limit to 512M and 1G just to be sure

It would always boot to not monitored on restart following above steps each time on the new image.

Maybe I just need to add `gonit start to a boot up file?? Where should I do that?

Looking forward to anyone's ideas they might have.

Ibone 2020-12-16 12:33:55 UTC #10

Hi @lets4done,

we were able to reproduce, we'll investigate what's happening, and we will update this thread with more information.

Regards,
Ibone.

lets4done 2020-12-17 04:12:15 UTC #11

Ok thanks
Wheww keep me updated.

jariza 2020-12-17 09:46:01 UTC #12

Hi @lets4done

We reproduced it. It seems the command below is not being executed after initializaing the services:

$  sudo gonit monitor all

So.. Even the services are running (e.g. apache):

sudo ps aux | grep apache
root       618  0.0  0.4  14416  8176 ?        Ss   09:34   0:00 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
daemon     620  0.0  0.7 1867484 13860 ?       Sl   09:34   0:00 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
daemon     621  0.0  0.7 1867484 13860 ?       Sl   09:34   0:00 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
daemon     622  0.0  0.7 1867484 13860 ?       Sl   09:34   0:00 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf

The ctlscritp.sh and gonit commands say they're stopped.

I opened a task to fix the issue. In the meantime, please manually execute sudo gonit monitor all after rebooting your cloud instance.

Best regards,

Juan Ariza

marcos 2020-12-17 13:03:13 UTC #13

Hi @lets4done, in the meantime we've identified a patch to solve your issues. Please run the following command (copy and paste all lines at once to your console):

sudo sudo patch -p0 -d / <<"EOF"
--- /opt/bitnami/nami/node_modules/provisioner/lib/recipes/gonit.js 2020-12-17 12:55:34.000000000 +0000
+++ /opt/bitnami/nami/node_modules/provisioner/lib/recipes/gonit.js 2020-12-17 12:55:38.484267825 +0000
@@ -79,6 +79,7 @@
         return __awaiter(this, void 0, void 0, function* () {
             logger.info("Starting gonit monitoring service");
             $os.runProgram(gonitBinary, []);
+            $os.runProgram("sleep", "2");
             logger.info("Start monitoring all services from gonit");
             yield utils.retry(() => {
                 $os.runProgram(gonitBinary, ["monitor", "all"]);
EOF

Afterwards you should be able to reboot and Gonit monitoring will work without issues.

Unable to perform start operation Export start for apache failed with exit code 1

lets4done 2020-12-18 05:43:46 UTC #14

That patch seems to have fixed it. Not thoroughly tested but does seem to work on two images I have here which it was not doing before. Restarted once and worked. So Thanks https://community.bitnami.com/users/marcos and Bitnami team. appreciate the use of your images.

jariza 2020-12-18 07:58:06 UTC #15

Glad to hear that it worked @lets4done

We'll let you know it when we release new images not affected by this issue with gonit.

system 2021-01-01 07:58:08 UTC #16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!