Azure RabbitMQ Cluster - Nodes unable to join cluster after re-IP of the VMs

jsantos 2019-04-24 17:42:35 UTC #1

Keywords: RabbitMQ - Linux - Technical issue - Other
bnsupport ID: ffbb2ac9-91b4-a059-9d10-5628680fd629
Description:
This cluster was deployed through the Azure Marketplace.

Had to re-IP all 3 VMs of my RabbitMQ cluster consisting of a master and 2 slave nodes. The master came up fine after a required reboot after the IP change. The 2 slave nodes did not rejoin the cluster after the required reboot after the IP change.

prodrabbitmq10 - master
prodrabbitmq11 - slave
prodrabbitmq12 - slave

I attempted to start the app on prodrabbitmq11 and received the following output:

DIAGNOSTICS
===========

attempted to contact: [rabbit@prodrabbitmq11]

rabbit@prodrabbitmq11:
  * connected to epmd (port 4369) on prodrabbitmq11
  * epmd reports: node 'rabbit' not running at all
                  no other nodes on prodrabbitmq11
  * suggestion: start the node

jdrios 2019-04-25 10:12:37 UTC #2

Hi @jsantos

Could you share the content of the /etc/hosts file of each node? In case you find that there are outdated entries there, please update them to use the new IPs of the nodes.

Also, could you check the cluster status in each node?

sudo rabbitmqctl cluster_status

Was my answer helpful? Click on

jsantos 2019-05-02 20:28:33 UTC #3

Hello jdrios! Thank you for your assistance.

prodrabbitmq10:

bitnami@prodrabbitmq10:~$ hostname
prodrabbitmq10

bitnami@prodrabbitmq10:~$ cat /etc/hosts
127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters


# BEGIN PROVISIONER MANAGED HOST RECORDS
10.0.0.6 provisioner-local

# END PROVISIONER MANAGED HOST RECORDS

bitnami@prodrabbitmq10:/$ sudo rabbitmqctl cluster_status
[{nodes,[{disc,[rabbit@prodrabbitmq10,rabbit@prodrabbitmq11,
                rabbit@prodrabbitmq12]}]},
 {running_nodes,[rabbit@prodrabbitmq10]},
 {cluster_name,<<"rabbit@prodrabbitmq10">>},
 {partitions,[]},
 {alarms,[{rabbit@prodrabbitmq10,[]}]}]

prodrabbitmq11:

bitnami@prodrabbitmq11:/$ hostname
prodrabbitmq11

bitnami@prodrabbitmq11:/$ cat /etc/hosts
127.0.0.1       localhost
10.4.13.19      rabbit@prodrabbitmq11
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters


# BEGIN PROVISIONER MANAGED HOST RECORDS
10.0.0.6 provisioner-peer
10.0.0.4 provisioner-local

# END PROVISIONER MANAGED HOST RECORDS

bitnami@prodrabbitmq11:/$ sudo rabbitmqctl cluster_status
sudo: unable to resolve host prodrabbitmq11: Connection timed out
warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)
Error: unable to perform an operation on node 'rabbit@prodrabbitmq11'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@prodrabbitmq11
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: [rabbit@prodrabbitmq11]

rabbit@prodrabbitmq11:
  * connected to epmd (port 4369) on prodrabbitmq11
  * epmd reports: node 'rabbit' not running at all
                  no other nodes on prodrabbitmq11
  * suggestion: start the node

Current node details:
 * node name: 'rabbitmqcli-16467-rabbit@prodrabbitmq11'
 * effective user's home directory: /opt/bitnami/rabbitmq/.rabbitmq
 * Erlang cookie hash: 0cO6nNx0UEr8HpJAPJpcXQ==

prodrabbitmq12:

bitnami@prodrabbitmq12:/$ hostname
prodrabbitmq12

bitnami@prodrabbitmq12:/$ cat /etc/hosts
127.0.0.1       localhost
10.4.13.21      rabbit@prodrabbitmq12
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters


# BEGIN PROVISIONER MANAGED HOST RECORDS
10.0.0.6 provisioner-peer
10.0.0.5 provisioner-local

# END PROVISIONER MANAGED HOST RECORDS

bitnami@prodrabbitmq12:/$ sudo rabbitmqctl cluster_status
sudo: unable to resolve host prodrabbitmq12: Connection timed out
[sudo] password for bitnami:
warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)
Error: unable to perform an operation on node 'rabbit@prodrabbitmq12'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@prodrabbitmq12
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: [rabbit@prodrabbitmq12]

rabbit@prodrabbitmq12:
  * connected to epmd (port 4369) on prodrabbitmq12
  * epmd reports: node 'rabbit' not running at all
                  no other nodes on prodrabbitmq12
  * suggestion: start the node

Current node details:
 * node name: 'rabbitmqcli-16056-rabbit@prodrabbitmq12'
 * effective user's home directory: /opt/bitnami/rabbitmq/.rabbitmq
 * Erlang cookie hash: 0cO6nNx0UEr8HpJAPJpcXQ=

@jdrios - Some additional information for you. In the /etc/hosts section for the the 2 slave nodes, I added the line that specifies the IP address with the hostname as a forum I looked at suggested adding that entry as a fix.

Please let me know if you need additional information and again thank you for your assistance.

Jeremy

jdrios 2019-05-03 10:38:23 UTC #4

Hi @jsantos

I have notified the team about this issue and they will provide further assistance.

Was my answer helpful? Click on

abjimenez 2019-05-07 07:57:31 UTC #5

Hi @jsantos,

Looking at the message that rabbitmqctl cluster_status reports:

$ sudo rabbitmqctl cluster_status
sudo: unable to resolve host prodrabbitmq11: Connection timed out

$ sudo rabbitmqctl cluster_status
sudo: unable to resolve host prodrabbitmq12: Connection timed out

Can you try to resolve the ip for the above addresses? You can use the following command that you need to execute in each of the VMs:

ping -c1 prodrabbitmq11

ping -c1 prodrabbitmq12

Azure has an internal DNS server that you may want to use to make each of the machine hostnames resolvable. You can access to it through Resource group > Network interface > DNS servers:

Maybe after the change in the private ips of the nodes, this service is not behaving correctly. To workaround it, as you mentioned, you can use the /etc/hosts file. However, it seems you added the complete identifier of the rabbitmq node <node>@<machine> instead of the machine only:

$ cat /etc/hosts
...
10.4.13.19      rabbit@prodrabbitmq11

Can you change the /etc/hosts file to make sure it contains the following entries?:

Note that you will need to substitute IP_OF_NODE_0 to the real value of the IP.

Node prodrabbitmq10:

IP_OF_NODE_0  prodrabbitmq10
10.4.13.19    prodrabbitmq11
10.4.13.21    prodrabbitmq12

IP_OF_NODE_0  provisioner-local

Node prodrabbitmq11:

IP_OF_NODE_0  prodrabbitmq10
10.4.13.19    prodrabbitmq11
10.4.13.21    prodrabbitmq12

IP_OF_NODE_0  provisioner-peer
10.4.13.19    provisioner-local

Node prodrabbitmq12:

IP_OF_NODE_0  prodrabbitmq10
10.4.13.19    prodrabbitmq11
10.4.13.21    prodrabbitmq12

IP_OF_NODE_0  provisioner-peer
10.4.13.21    provisioner-local

After those changes, try to resolve the names again with the ping command and restart the rabbitmq services in all the VMs:

sudo service bitnami restart

Let us know if it works.

Regards,
Andrés Bono.

system 2019-05-21 07:57:37 UTC #6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

jota 2019-05-22 06:36:33 UTC #7

jsantos 2019-05-22 06:37:31 UTC #8

Andres,

Apologies for the extremely long delay in getting back to you. I was finally able to go through your recommended changes to the /etc/hosts files and restart the Bitnami service. I am able to ping the slave nodes from the master but I cannot ping the master from either of the slave nodes. But the good news is that I am able to see the nodes in the RabbitMQ management panel:

I will test the queues and let you know if they work or not as well as looking into why I can only ping successfully from the master node.

Thanks again for your help,

Jeremy

Bitnami provides free all-in-one installers, virtual machines and cloud images for popular open source applications. Get them now!