Rabbitmqctl, rabbitmq not working anymore on bash/cli (Azure Marketplace deploy)

:warning: IMPORTANT, please fill the questions

We assume you are using Bitnami to deploy your application.

  • Which version of the application are you using?:
    RabbitMQ: 3.8.3
    Erlang: 22.2

  • Please choose how you got the application: Installer (Windows, Linux, macOS), cloud image (AWS, GCE, Azure, …) or VM (VMDK, VBOX):
    Cloud Image from Azure Marketplace (hosted on Azure), cluster with 2 nodes

  • Have you installed any plugin or modified any configuration file?:
    Configured SSL by modifying ‘/opt/bitnami/rabbitmq/conf/rabbitmq.config’ with:

[
 {rabbit,
  [
   {tcp_listeners, [5672]},
   {ssl_listeners, [5671]},
   {ssl_options,[
      {cacertfile, "/opt/bitnami/rabbitmq/certificates/root.crt"},
      {certfile, "/opt/bitnami/rabbitmq/certificates/public.crt"},
      {keyfile, "/opt/bitnami/rabbitmq/certificates/private.key"},
      {verify, verify_peer},
      {fail_if_no_peer_cert, false}
   ]},
   {disk_free_limit, {mem_relative, 1.0} },
   {cluster_partition_handling, ignore },
   {default_vhost, <<"/">>},
   {default_user, <<"user">>},
   {default_permissions, [<<".*">>, <<".*">>, <<".*">>]}
  ]
 },
 {rabbitmq_management,
  [
   {listener, [{port, 15672 }, {ip, "0.0.0.0"}]}
  ]
 }
].

=> Not working and unable to troubleshoot, due to issue noted below

  • Describe here your question/suggestion/issue (expected and actual results):
    running ‘sudo rabbitmqctl’ or ‘sudo rabbitmq-plugins’ produces an error;
    bitnami@testrabbitmq1:~$ sudo rabbitmqctl
    [sudo] password for bitnami:
    escript: exception error: undefined function rabbitmqctl_escript:main/1
    in function escript:run/2 (escript.erl, line 757)
    in call from escript:start/1 (escript.erl, line 277)
    in call from init:start_em/1
    in call from init:do_boot/3

bitnami@testrabbitmq1:~$ sudo rabbitmq-plugins list
escript: exception error: undefined function rabbitmqctl_escript:main/1
in function escript:run/2 (escript.erl, line 757)
in call from escript:start/1 (escript.erl, line 277)
in call from init:start_em/1
in call from init:do_boot/3

  • Steps to reproduce the issue (if relevant):
    running ‘sudo rabbitmqctl’ or ‘sudo rabbitmq-plugins’

  • Copy the apache log (if relevant):
    Not relevant

Hi @brandonwinten,

We have a Support Tool that will gather relevant information for us to analyze your configuration and logs. Could you please execute it on the machine where the stack is running by following the steps described in the guide below?

Please note that you need to paste the code ID that is shown at the end.

Hi @gongomgra

Just collected the log files through the support tool.
Code ID, which was provided: 90a7fc3a-2ae4-7e19-7d03-bcb6f3c4ab32

Thanks in advance

Sincerely
Brandon

Hi @brandonwinten,

I don’t see any error in the RabbitMQ logfile. What I see is that the SSL certificates you are using don’t belong to the rabbitmq user that is running the RabbitMQ service. Can you change the ownership to check if it helps to fix the issue?

sudo chown rabbitmq /opt/bitnami/rabbitmq/certificates/root.crt
sudo chown rabbitmq /opt/bitnami/rabbitmq/certificates/public.crt
sudo chown rabbitmq /opt/bitnami/rabbitmq/certificates/private.key

Hi @gongomgra

Applied the changes, restarted the service and this had no effect.
Even after a reboot the same error persists.

I cannot perform any rabbitmqctl or rabbitmq-plugins command.
When not using sudo in the command I get the following error (and warning);
warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running “locale” in your shell)

Error when reading /opt/bitnami/rabbitmq/.rabbitmq/.erlang.cookie: eacces

What could be the issue?

Thanks in advance

Sincerely
brandon

Hi @brandonwinten,

Thanks for the info. I think it has something to do with your locale settings

Can you check your settings?

locale

According to the error message you should configure it to UTF-8. Can you check your available locales?

locale -a

Hi @gongomgra

This is the output when I run ‘locale’;
bitnami@testrabbitmq0:~$ locale
LANG=
LANGUAGE=
LC_CTYPE=“POSIX”
LC_NUMERIC=“POSIX”
LC_TIME=“POSIX”
LC_COLLATE=“POSIX”
LC_MONETARY=“POSIX”
LC_MESSAGES=“POSIX”
LC_PAPER=“POSIX”
LC_NAME=“POSIX”
LC_ADDRESS=“POSIX”
LC_TELEPHONE=“POSIX”
LC_MEASUREMENT=“POSIX”
LC_IDENTIFICATION=“POSIX”
LC_ALL=

When I run ‘locale -a’ I get a list that’s too long to paste here, though UTF8 is a lot of times in it but not as ‘UTF8’, but for every language separately, although there is one exception ‘C.UTF8’.

Could this be the problem?
Because in our production environment I checked and get the same warning and error when not using ‘sudo’ but everything works there. I checked the ‘locale’ there as well and this is the same.

Thanks

Sincerely
Brandon

Hi @brandonwinten,

I think using UTF8 or UTF-8 in the LANG or LC_* definitions don’t make any difference if the exists. I mean, a locale can be named C.UTF8 while other can be (it’s an example, not sure if the actual value) en_US.UTF-8. Can you try running the command below that will temporary set the locale to a different value for testing?

LANG="C.UTF8" LC_ALL="C.UTF8" rabbitmqctl version

If the above work, can you try running other commands as well? If you need to use sudo, please run the command as follows

sudo LANG="C.UTF8" LC_ALL="C.UTF8" rabbitmqctl version

Hi @gongomgra

Same result unfortunately.
Could it be that rabbitmqctl_escript needs to de recompiled or that ecript cannot find rabbitmqctl?
I’m just thinking out loud, been searching for a solution for 2 weeks now (also before opening a thread in here).
To be fair, this makes me nervous about our production environment.

Thanks

sincerely
Brandon

Hi @brandonwinten,

I launched a fresh new RabbitMQ Cluster with two nodes using the Azure Marketplace launcher and I couldn’t reproduce your issue (but I think I reproduced it partially). I launched it in the South Central US region. Can you tell us in which region are you launching your cluster?

In the cluster I launched, the default locale is set to en_US.UTF-8, and the rabbitmqctl command is working fine.

$ sudo rabbitmqctl version
3.8.5

$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

I reproduced the warning message you are getting when using the command I gave you previously setting LANG="C.UTF8", but the RabbitMQ version was given as a result of running that command

$ sudo LANG="C.UTF8" LC_ALL="C.UTF8" rabbitmqctl version
warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)
3.8.5

Can you try setting the en_US.UTF-8 locale?

sudo LANG="en_US.UTF-8" LC_ALL="en_US.UTF-8" rabbitmqctl version

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.