autocluster and prometheus_rabbitmq_exporter on kubernetes make rabbitmq segfault

[janwillies]

I'm using rabbitmq-3.6.6 (from dockerhub) with the following plugins:

autocluster-0.6.1.ez (from master)
rabbitmq_aws-0.1.3.ez (from master)
prometheus-3.0.1.ez
prometheus_process_collector-1.0.0.ez
prometheus_rabbitmq_exporter-v3.6.5.6.ez

As soon as I hit /api/metrics rabbitmq segfaults. Platform is kubernetes. I'm not sure which faults it is, just opening the bug here in case anyone runs into the same problem

           PID: 24250 (beam.smp)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Thu 2016-12-01 14:09:42 UTC (2min 49s ago)
  Command Line: /usr/lib/erlang/erts-8.0.5/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.6/ebin -noshell -noinput -s rabbit boot -name rabbit@10.1.81.7 -boot start_sasl -config /etc/rabbitmq-sxris/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger tty -rabbit error_logger tty -rabbit sasl_error_logger tty -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.6/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@10.1.81.7-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@10.1.81.7" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
    Executable: /usr/lib/erlang/erts-8.0.5/bin/beam.smp
 Control Group: /
         Slice: -.slice
       Boot ID: 3b17e7f05a894cc0b77331744b33a795
    Machine ID: 1a8d01d73f6744fc858f25a25be315f8
      Hostname: ip-10-43-128-107.eu-central-1.compute.internal
      Coredump: /var/lib/systemd/coredump/core.beam\x2esmp.0.3b17e7f05a894cc0b77331744b33a795.24250.1480601382000000000000.xz
       Message: Process 24250 (beam.smp) of user 0 dumped core.

How can I help solving the issue?

[michaelklishin]

RabbitMQ has no native code and neither do those two plugins. Switch to a more recent Erlang release and make sure it's 64 bit.

[deadtrickster]

Hello,

Try to disable prometheus_process_collector. It actually has native code https://github.com/deadtrickster/prometheus_process_collector/blob/master/c_src/prometheus_process_collector_nif.c.

From what people reported so far: different glibc version and/or not x64.

Please let me know the result.

[janwillies]

Thanks @deadtrickster, deactivating prometheus_process_collector fixes the problem.

I'll try to build the prometheus_process_collector-plugin with the same distro which I'm running the resulting rabbitmq next to rule out glibc mismatches

[jperville]

I ran into this issue on some of my servers. This is strange because all servers are running the same version of RabbitMQ (3.6.8) and this plugin (3.6.8.1). I packaged everything in my own docker image, based on the official rabbitmq:3.6.8 image. The servers on which I have this issue are bare metal running Centos 7.3 64 bits, while those where it work are VMs running Centos 7.4 64 bits.

My workaround for now is to disable the prometheus_process_collector plugin.

[deadtrickster]

what glibc versions on build machine and on host?

[jperville]

Using pre-built binary of prometheus_process_collector from https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases

On the host which builds the docker image (ubuntu 14.04 64 bits):

root@ci:~# dpkg -l | grep libc6
ii  libc6:amd64                        2.19-0ubuntu6.11                  amd64        Embedded GNU C Library: Shared libraries
ii  libc6-dev:amd64                    2.19-0ubuntu6.11                  amd64        Embedded GNU C Library: Development Libraries and Header Files

Inside the docker image which uses the pre-built binary (docker image rabbitmq:3.6.8, based on debian:jessie):

$ docker run --rm -ti rabbitmq:3.6.8 /bin/bash
root@55b9f32cf688:/#  dpkg -l | grep libc6
ii  libc6:amd64              2.19-18+deb8u7           amd64        GNU C Library: Shared libraries

On the production hosts which use the image (Centos 7.3 64 bits):

# VM server, not crash
[root@first ~]# rpm -aq | grep glibc
glibc-2.17-105.el7.x86_64
glibc-common-2.17-105.el7.x86_64
# real server, crash
[root@second~]# rpm -aq | grep glibc
glibc-2.17-157.el7_3.1.x86_64
glibc-common-2.17-157.el7_3.1.x86_64

[deadtrickster]

And what other server (that do not crash) have?

[deadtrickster]

Thanks a lot! Could you please try to rebuild process collector on machine that crashes and try this version in your docker image?

[jperville]

Hi @deadtrickster , I investigated my issue and actually rabbitmq crashed not because of some binary incompatibility in the prometheus_process_collector module as described in this issue but because of selinux issues (it tried to load the *.so from the process collector plugin and selinux denied the request). I solved the issue by overriding the RABBITMQ_PLUGINS_EXPAND_DIR environment variable to point outside of my data directory (eg. to /tmp/rabbitmqplugin-expand).

[deadtrickster]

Sounds crazy :-) Could you please briefly describe it and submit PR to README?

[jperville]

I will submit a kubernetes manifest which displays the problem if you are curious.

[deadtrickster]

thanks