Ctlscript.sh missing from PostgreSQL with Replication

Keywords: PostgreSQL - Google Cloud Platform - Technical issue - Other
bnsupport ID: cc364d51-ffc8-ad0c-55f1-b15172c9575e
Description:
I have installed PostgreSQL with Replication with one primary and one replica via the GCP marketplace at https://console.cloud.google.com/marketplace/details/bitnami-launchpad/postgresql-cluster

In the documentation it says :

https://docs.bitnami.com/general/infrastructure/postgresql/administration/control-services/

Obtain the status of a service:
sudo /opt/bitnami/ctlscript.sh status

But there is no such script in the /opt/bitnami directory

root@postgresql-cluster-1-node-0:/bitnami/postgresql/conf/conf.d# ls -alh /opt/bitnami
total 60K
drwxr-xr-x 11 root    root    4.0K Aug 17 09:04 .
drwxr-xr-x  3 root    root    4.0K Aug 12 19:02 ..
drwxr-xr-x  3 root    root    4.0K Jun 25 18:48 bnsupport
lrwxrwxrwx  1 root    root      52 Jun 25 18:10 bnsupport-tool -> /opt/bitnami/bnsupport/bnsupport-0.9.2-linux-x64.run
drwxr-xr-x  5 root    root    4.0K Jul 29 09:35 common
-rw-r--r--  1 root    root       4 Aug 17 09:04 .firstboot.status
drwxr-xr-x  4 root    root    4.0K Jun 25 15:42 gonit
drwxr-xr-x  3 root    root     12K Aug 10 14:54 licenses
drwxr-xr-x  7 root    root    4.0K Aug 10 14:51 nami
drwxr-xr-x 10 root    root    4.0K Aug 12 19:05 postgresql
-rw-r--r--  1 root    root      89 Aug 17 09:04 properties.ini
drwxr-xr-x  3 root    root    4.0K Aug 11 14:46 scripts
drwxr-xr-x  2 bitnami bitnami 4.0K Aug 18 12:04 stats
drwxr-xr-x  3 root    root    4.0K Aug 12 19:05 var

So in despair I tried to do systemctl restart bitnami (because I add changed some configuration in postgresql.conf) but the bitnami service somehow tried to fire a secondary postgres instance using the same datadir and now my postgres logs are flooded with the error :

2020-08-18 12:22:27.976 GMT [1283] FATAL:  the database system is shutting down

My applications accessing postgresql can somehow still work properly. There seems to be a “ghost” postgres process fired in duplicated by the bitnami service.

So my questions are :

  • how is it that I don’t have a ctlscript.sh like mentionned in the documentation ?
  • why did systemctl restart bitnami resulted in this disaster ?
  • what do I need to do to get this ctlscript ? Can I download it somewhere ?

Thanks

I managed to kill the zombie processed with

systemctl kill bitnami
systemctl start bitnami

But I still want to now why there is no ctlscript and why the restart command I typed provoked this.

I have found this in the server logs during startup

postgres: could not write external PID file "/opt/bitnami/postgresql/tmp/postgresql.pid": Permission denied

Maybe it explains why the server cannot be stopped properly if the stop script is based on that pid file that could not be created during startup ?

Hi @val789,

I checked the bnsupport bundle and I see the file is owned by root.

postgresql/tmp:
total 12
drwxrwxr-x  2 postgres postgres 4096 Aug 17 09:04 .
drwxr-xr-x 10 root     root     4096 Aug 12 19:05 ..
-rw-r--r--  1 root     root        5 Aug 17 09:04 postgresql.pid

Can you run the next command to set it to postgres and try again?

sudo chown postgres /opt/bitnami/postgresql/tmp/postgresql.pid

Apart from that, the ctlscript.sh is not shipped in the Multi-tier assets. You need to use the service command as mentioned in this guide

https://docs.bitnami.com/google-templates/infrastructure/mariadb/administration/control-services/

Thank you for you answer and the time you took to investigate my issue.

I have some further issues :

Thank you

Hi @val789,

Can you tell us where did you get that link to our documentation? I visted the webpage for the PostgreSQL with replication in the Google Marketplace and it is pointing to docs.bitnami.com/google-templates as shown in the next screenshot

Sorry for the wrong link in my latest message, it is the one for PostgreSQL

https://docs.bitnami.com/google-templates/infrastructure/postgresql/administration/control-services/

Regarding the permissions issue, I launched a fresh new server and I verified the file is owned by root by default and PostgreSQL works fine

Linux gonzalo-postgresql-cluster-1-node-0 4.19.0-10-cloud-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
       ___ _ _                   _
      | _ |_) |_ _ _  __ _ _ __ (_)
      | _ \ |  _| ' \/ _` | '  \| |
      |___/_|\__|_|_|\__,_|_|_|_|_|

************************************************************
  This is a Bitnami server.


************************************************************
bitnami@gonzalo-postgresql-cluster-1-node-0:~$ ps aux | grep -i postgre
postgres  1633  0.0  1.6 943980 61332 ?        S    08:26   0:00 /opt/bitnami/postgresql/bin/postgres -D /bitnami/postgresql/data --config-file=/opt/bitnami/postgresql/conf/postgresql.conf --external_pid_file=/opt/bitnami/postgresql/tmp/postgresql.pid --hba_file=/opt/bitnami/postgresql/conf/pg_hba.conf
postgres  1663  0.0  0.3 944220 13712 ?        Ss   08:26   0:00 postgres: checkpointer
postgres  1664  0.0  0.2 944188  9392 ?        Ss   08:26   0:00 postgres: background writer
postgres  1665  0.0  0.5 943980 22184 ?        Ss   08:26   0:00 postgres: walwriter
postgres  1666  0.0  0.2 944800  8388 ?        Ss   08:26   0:00 postgres: autovacuum launcher
postgres  1667  0.0  0.1  66040  4456 ?        Ss   08:26   0:00 postgres: stats collector
postgres  1668  0.0  0.1 944752  6572 ?        Ss   08:26   0:00 postgres: logical replication launcher
postgres  2428  0.0  0.2 945328 10556 ?        Ss   08:27   0:00 postgres: walsender replication 10.128.0.84(36850) streaming 0/4000060
postgres  2429  0.0  0.2 944936 10296 ?        Ss   08:27   0:00 postgres: walsender replication 10.128.0.85(49956) streaming 0/4000060
bitnami   2459  0.0  0.0   6076   880 pts/0    S+   08:32   0:00 grep -i postgre
bitnami@gonzalo-postgresql-cluster-1-node-0:~$ ls -lart /opt/bitnami/postgresql/tmp/postgresql.pid
-rw-r--r-- 1 root root 5 Aug 20 08:26 /opt/bitnami/postgresql/tmp/postgresql.pid
bitnami@gonzalo-postgresql-cluster-1-node-0:~$ postgres --version
postgres (PostgreSQL) 12.4
bitnami@gonzalo-postgresql-cluster-1-node-0:~$

Did you modify anything in the PostgreSQL or the server configuration before having this issue? Which version of PostgreSQL are you running?

The website won’t let me put more than one link in each message so please see other messages below for images and links.

You are right for the three links at the bottom in your screenshots which lead to the appropriate page.
However I initially followed the very first documentation link on the page :

SEE IMAGE 1

Then “Documentation” on the right :

SEE IMAGE 2 BELOW

Then “Start or Stop Services” on the right :

SEE IMAGE 3 BELOW

Which leads to :

SEE IMAGE 4 BELOW

IMHO thelanding documentation page at https://bitnami.com/stack/postgresql is very confusing because there are multiple deployment offering and it’s not obvious at first that there are entirely different and incompatible documentation pages for the different deployments methods. This should be state more clearly upfront. Also when coming from the multi tier deployment from the GCP marketplace the “Multi-Tier” button should be preselected and the litle “Documentation” link for GCP and Azure hidden at the bottom should also be straighly visible in the “Additional Ressources” likefor the other deployment offerings :

SEE IMAGE 5 BELOW

As for the postgresqlSQL instance, no I didn’t modifying anything before having this issue. I am using a freshly deployed instance at 17.08.2020, 11:03:34 by bitnami which included postgresql version 12.3

IMAGE 1

IMAGE 2

IMAGE 3

IMAGE 4

IMAGE 5

Hi @val789,

Thanks for the detailed info. The main PostgreSQL webpage on our website (https://bitnami.com/stack/postgresql) points to more generic documentation we have for the product as it may be useful for most of our offerings (installers, virtual machines, and single-tier cloud images). If you click the “Multi-Tier” button on that page, you will get a new set of links pointing to the specific multi-tier documentation.

I will contact the docs team to check if we can update the link in the Google Marketplace to https://bitnami.com/stack/postgresql/multitier for this specific product.

Regarding the starting services issue, can you try the latest 12.4 version? Maybe the engineering team did any change when updating the version to fix that issue. I couldn’t reproduce it in PostgreSQL 12.4 as mentioned in my previous post.

Ok thank you for your answer.
I can try with deploying another cluster with the latest version available on the marketplace. However I can’t find in the documentation the recommended way to upgrade an existing cluster to the latest bitnami stack version. How do I upgrade it ? Thanks

I can confirm there was definitely a bug in the stack where I installed it.
I installed it again today and the service commands work properly.
On the older stack installation, the service bitnami stop command blocks for like 2 minutes then returns but doesn’t stop the postgres process.

I still need help for how to upgrade my first cluster where the bugs exist. How do I upgrade the stack to the latest bitnami stack installation ?

Hi @val789,

Thanks for letting us know. I’m glad the new version worked for you. Unfortunately, upgrading components is not possible. You will need to launch a new cluster and migrate the database information as explained in the guide below

https://docs.bitnami.com/google-templates/infrastructure/postgresql/administration/backup-restore-postgresql/

Ok I understand.
Can you instead link me to a diff or tell me what file I need to change to make the service stop command works, regarding of what was actually changed between these two releases ?

Hi @val789,

I’m afraid I don’t know what was the change or if it is possible to implement it in your old version instance, but we highly recommend you to keep using the latest version (12.4) for performance and security reasons.

I understand the need to keep the latest version, but without a supported and documented way to upgrade the stack, it would take a lot of maintenance to migrate the cluster every time a new version is released. IMHO bitnami should provide an upgrade script and documentation. Would you like to consider this feature request ?