RabbitMQ Clustering on CentOS 6.2

RabbitMQ takes a little extra effort to get it working on CentOS 6, particularly if using clustering.  After you’ve installed, you need to do the following before trying to set up the cluster.

Edit /etc/hosts to ensure all cluster nodes can resolve the hostnames of each other.  These hostnames need to match what is displayed after `rabbit@` in rabbitmqctl status.

Next, you need to take care of the ports needed for distributed Erlang processes to communicate. If you don’t do this, then no matter what you do, when you try to create the cluster it will give you ‘error_no_running_cluster_nodes’.

RabbitMQ runs on Erlang, and tor the nodes to talk, they need a few extra ports open beyond the standard 5672 that AMQP uses.  One is for epmd (Erlang Port Mapper Daemon) and that is a standard 4369.  Then you have a port range that Erlang nodes to communicate. You need to define this range so you can open these ports on your firewall.

To tell RabbitMQ to instruct Erlang to communicate on a certain range of ports, create a file at /etc/rabbitmq/rabbitmq.config with the following contents:

[
{kernel, [{inet_dist_listen_min, 9100},{inet_dist_listen_max, 9105}]}
].

This will force the nodes in the cluster to only communicate over these ports.  Restart RabbitMQ with `service rabbitmq-server restart` so the change takes effect.  Now for the iptables configuration.  Update /etc/sysconfig/iptables with the following rules:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 5672 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 4369 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9100:9105 -j ACCEPT

After those changes, reload iptables with `/etc/init.d/iptables restart` so the new rules take effect.  At this point, you should be good to follow the RabbitMQ clustering guide step by step.

Closing orphaned RabbitMQ connections

I’ve been working with RabbitMQ a lot lately, and one problem I’ve run into is that occasionally during the course of development, something goes wrong and we leave a lot of connections open.  Enabling the heartbeat will take care of this for the most part, but if someone didn’t do that, you can end up with connections left open on the server.  Here is a quick shell script to close all the connections from a particular IP address:

#!/bin/sh
peerToDisconnect=1.2.3.4
for c in `rabbitmqctl list_connections peer_address pid | grep $peerToDisconnect`;
do
if [ $c != $peerToDisconnect ];
then rabbitmqctl close_connection $c "Orphaned - disconnecting";
fi;
done