Node cannot join existing cluster

[hac3ru]

Describe the bug

I had a 3 node cluster. Had to reinstall one of the VMs, I ran rabbitmqctl stop_app, I ran the cluster_forget_node on the remaining nodes and reinstalled the VM. Same hostname, same IP address, everything.
I reinstalled RabbitMQ on the machine, copied the erlang cookie, copied the config file, started the service and ran

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@rabbitmq-01

but I'm getting:

2025-10-09 16:53:55.755705+02:00 [notice] <0.7042.0> RabbitMQ metadata store: candidate -> leader in term: 4 machine version: 1, last applied 4
2025-10-09 16:53:55.765637+02:00 [notice] <0.7005.0> Feature flags: checking nodes `rabbit@rabbitmq-02` and `rabbit@rabbitmq-01` compatibility...
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags: error while running:
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags:   rabbit_ff_registry_wrapper:inventory[]
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags: on node `rabbit@rabbitmq-03`:
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags:   exception error: {erpc,timeout}
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags:     in function  erpc:call/5 (erpc.erl, line 1380)
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags:     in call from rabbit_ff_controller:rpc_call/5 (src/rabbit_ff_controller.erl, line 1613)
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags:     in call from rabbit_ff_controller:'-rpc_calls/5-fun-1-'/7 (src/rabbit_ff_controller.erl, line 1645)
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0>
2025-10-09 16:54:55.767714+02:00 [warning] <0.7005.0> Feature flags: error while collecting inventory from nodes ['rabbit@rabbitmq-01','rabbit@rabbitmq-02','rabbit@rabbitmq-03']:
2025-10-09 16:54:55.767714+02:00 [warning] <0.7005.0> {error,{erpc,timeout}}
2025-10-09 16:54:57.617301+02:00 [error] <0.6906.0> ** Node 'rabbit@rabbitmq-03' not responding **
2025-10-09 16:54:57.617301+02:00 [error] <0.6906.0> ** Removing (timedout) connection **
2025-10-09 16:54:57.617301+02:00 [error] <0.6906.0>
2025-10-09 16:54:57.637616+02:00 [info] <0.7005.0>
2025-10-09 16:54:57.637616+02:00 [info] <0.7005.0>  Starting RabbitMQ 4.1.4 on Erlang 27.3.4.2 [jit]
2025-10-09 16:54:57.637616+02:00 [info] <0.7005.0>  Copyright (c) 2007-2025 Broadcom Inc and/or its subsidiaries
2025-10-09 16:54:57.637616+02:00 [info] <0.7005.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>  node           : rabbit@rabbitmq-02
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>  home dir       : /var/lib/rabbitmq
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>  config file(s) : /etc/rabbitmq/rabbitmq.conf
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>  cookie hash    : UvHVskhv44N7QwUVknDo1w==
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>  log(s)         : /var/log/rabbitmq/rabbit.log
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>                 : <stdout>
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>  data dir       : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-02
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags: error while running:
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags:   rabbit_ff_registry_wrapper:inventory[]
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags: on node `rabbit@rabbitmq-03`:
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags:   exception error: {erpc,timeout}
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags:     in function  erpc:call/5 (erpc.erl, line 1380)
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags:     in call from rabbit_ff_controller:rpc_call/5 (src/rabbit_ff_controller.erl, line 1613)
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags:     in call from rabbit_ff_controller:'-rpc_calls/5-fun-1-'/7 (src/rabbit_ff_controller.erl, line 1645)
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0>
2025-10-09 16:56:12.622288+02:00 [error] <0.7111.0> ** Node 'rabbit@rabbitmq-03' not responding **
2025-10-09 16:56:12.622288+02:00 [error] <0.7111.0> ** Removing (timedout) connection **
2025-10-09 16:56:12.622288+02:00 [error] <0.7111.0>
2025-10-09 16:56:12.626374+02:00 [error] <0.7093.0> Mnesia('rabbit@rabbitmq-02'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit@rabbitmq-03'}
2025-10-09 16:56:12.626374+02:00 [error] <0.7093.0>
2025-10-09 16:56:12.629389+02:00 [notice] <0.45.0> Application mnesia exited with reason: stopped
2025-10-09 16:56:12.629529+02:00 [error] <0.7005.0>
2025-10-09 16:56:12.629529+02:00 [error] <0.7005.0> BOOT FAILED
2025-10-09 16:56:12.629529+02:00 [error] <0.7005.0> ===========
2025-10-09 16:56:12.629529+02:00 [error] <0.7005.0> Error during startup: {error,{erpc,timeout}}
2025-10-09 16:56:12.629529+02:00 [error] <0.7005.0>
2025-10-09 16:56:13.630349+02:00 [notice] <0.45.0> Application rabbit exited with reason: {{erpc,timeout},{rabbit,start,[normal,[]]}}
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>   crasher:
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     initial call: application_master:init/3
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     pid: <0.7004.0>
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     registered_name: []
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     exception exit: {{erpc,timeout},{rabbit,start,[normal,[]]}}
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>       in function  application_master:init/3 (application_master.erl, line 143)
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     ancestors: [application_controller,<0.10.0>]
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     message_queue_len: 1
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     messages: [{'EXIT',<0.7005.0>,normal}]
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     links: [<0.45.0>]
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     dictionary: []
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     trap_exit: true
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     status: running
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     heap_size: 2586
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     stack_size: 29
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     reductions: 76
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>   neighbours:
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>
2025-10-09 16:56:13.633831+02:00 [notice] <0.45.0> Application khepri exited with reason: stopped
2025-10-09 16:56:13.636447+02:00 [notice] <0.45.0> Application sysmon_handler exited with reason: stopped
2025-10-09 16:56:13.638643+02:00 [notice] <0.45.0> Application ra exited with reason: stopped
2025-10-09 16:56:13.639827+02:00 [notice] <0.45.0> Application os_mon exited with reason: stopped
2025-10-09 16:56:13.641014+02:00 [notice] <0.45.0> Application rabbitmq_prelaunch exited with reason: stopped

Somehow, it says that node3 is unresponsive, even though 1 and 3 are clustered and working. If I stop node 3, and try to join 2, it works. 1 and 2 can form the cluster, but the second I try to bring back node 3, I get the same error as above, but it's node2 who timesout.
What's going on?

Reproduction steps

1. Install a RabbitMQ cluster on 3 nodes (Ubuntu 24.04) via the RabbitMQ repos
2. Form a cluster - I used a config file to do it, I've only set the cluster name and the nodes.
3. Make sure all is healthy.
4. Remove a node from the cluster
5. Reinstall the OS on that node - and reinstall RabbitMQ. Give it the same name, same IP, etc.
6. Try to join the cluster on this node.

I was able to reproduce this three times already. As additional info, I'm running RabbitMQ 4.1.4-1

Expected behavior

The node should join the cluster, as any new node would.

Additional context

No response

[lukebakken]

I converted this to a discussion because there is no evidence this is a bug in RabbitMQ.

This is a decent start to a report, but what is missing are the exact commands, and transcripts of you running those commands, when you remove and reinstall RabbitMQ. There is a lot of information missing that you're asking us to guess.

In steps 4 and 5, what happens if you remove the node from the cluster, and uninstall RabbitMQ from that node (remove data directory, configuration). Then, re-install RabbitMQ and attempt to re-join the cluster. If that succeeds, then there must be something in how you're re-building the node that isn't quite identical to before.

[hac3ru]

Well, I can answer all of those, and I can even provide some sort of access to the test env, if you want it.

(will rerun this, and will also share a console output of removing the node + logs from the other nodes)
Removing the node was done via: systemctl stop rabbitmq-server on the node that was removed + rabbitmqctl forget_cluster_node rabbit@rabbitmq-03. That's all.
The VM was delete and a new one, from a template (different MAC address, different /etc/machine-id, etc) was created.
RabbitMQ was reinstalled as said, using the apt repo provided by RabbitMQ.

Steps 4 - 5 shouldn't matter, as that VM is completely deleted, and a new one is provisioned. Why would it matter, this is a new node, as far as the cluster knows, since the old rabbit@rabbitmq-03 was deleted, right?!

As I said, if you want, I can provide access to the cluster.

Also, the weird thing is that if I stop one of the nodes, node 3 can join. So this is between node 2 and node 3 in my setup. If Node1 is started from scratch, node 2 OR node 3 can join. If the cluster's already formed using node 1 and 2, 3 can't join. If the cluster's formed by 1 and 3, 2 can't join.

[lukebakken]

Also, the weird thing is that if I stop one of the nodes, node 3 can join

Yes, that is weird. If I can find the time, I'll try to reproduce this using docker.

If you could perform the steps that I asked, it would shine some light on this:

• Stop RabbitMQ on a node.
• Uninstall RabbitMQ and delete everything associated with it on that node.
• Use forget_cluster_node on another node to forget the one you just removed.
• Re-install, re-apply configuration, and re-start RabbitMQ on that node. It should re-join the cluster based on your configuration file.

[michaelklishin]

@hac3ru if a VM is deleted, chances are this is one of the two classic scenarios where a hostname changes between node starts or the data directory is removed.

In both cases the node can end up generating a different Erlang cookie (shared secret) locally, which means it won't be able to join a cluster that uses a different cookie, by design.

There is a distinct error message that will be logged in such cases by the existing cluster members such reset node tries to contact.

In case it is not obvious why a hostname change can have such an effect: by default, RabbitMQ node data directory path includes a hostname, for historical reasons and to make it possible to run multi-node integration tests on a single hosts easier.

[hac3ru]

Hello,

I am aware of the /var/lib/rabbitmq/.erlang_cookie file, and that has been copied over from one of the other nodes, and I ran chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang_cookie and chmod 0400 /var/lib/rabbitmq/.erlang_cookie
Indeed, the first time I did this, I got a message saying
Connection attempt from node 'rabbit-1939-2495@rabbitmq-02' rejected. Invalid challenge reply.

So the new node has nothing in /var/lib/rabbitmq/mnesia/ and the .erlang_cookie matches.