Disable metrics for some/all queues

[Thubo]

Hello there,

I'm trying to disable metrics collected for queues and exchanges (e.g. rabbitmq_queue_disk_reads).

Ideally I'm able to disable only for certain queues, however it would be alright to disable any queue related metrics.

(The reason is we have very high number of queues and the metrics take away significant performance from our brokers.)

So far I've tried:

[...]
{ prometheus, [
  { rabbitmq_exporter, [
    { path, "metrics" },
    { queue_messages_stat, [ ] },
    { exchange_messages_stat, [] }
  ] }
] },
[...]

Is there a simple way to disable any queue related metrics?

Thanks in advance for your help :)

Cheers,
Thubo

[deadtrickster]

Hi, thanks you for using this exporter. I hope can look into this at the beginning of the next week.

[Thubo]

Thanks a lot. If I can provide any help (logs, tests, ...) please let me know.

[deadtrickster]

Unfortunately this took longer than expected. Rescheduled to 3.6.8 release support

[deadtrickster]

application:set_env(prometheus, rabbitmq_exporter, [{exchange_messages_stat, [messages_published_total]}]). 

leaves only rabbitmq_exchange_messages_published_total.

example config

[
 {rabbit,
  [
   {loopback_users, []}
  ]},
 {prometheus,
  [{rabbitmq_exporter,
    [
     {queue_messages_stat, [messages_published_total]},
     {exchange_messages_stat, [messages_published_total]}
    ]}
  ]}
].

This however can't disable ALL queue metrics.
If you need this for performance reasons then rendering metrics isn't that costly at all. Enumerating and getting stat OTOH can be costly. So filtering by queue name can help here probably.
Getting queue info is a two-step process:

• rabbit_amqqueue:info_all(VHost);
• rabbit_mgmt_db:augment_queues(Queues, ...).

Not sure info_all can be removed (perhaps replaced with lighter version, but this is just an ETS query).
But augment_queues can be called only for required queues.

Can you give more details about your situation and how to reproduce?

[Thubo]

So, i've updated my RabbitMQ image to 3.6.8 (as well as your plugin).

Next enabled the rabbitmq_management plugin, but not your plugin.

If I use the configuration you posted above I run into the following error, as soon as I want to access any queue metrics (through the normale WebGUI). If I remove the config snippet, the WebGUI works as expected.

=ERROR REPORT==== 28-Mar-2017::07:28:37 ===
JSON encode error: {bad_term,{<<"x-priority">>,signedint,10}}
While encoding:
[{incoming,[]},
 {deliveries,[]},
 {messages_details,
     [{avg,0.0},
[...]

enabling the prometheus_rabbitmq_exporter plugin results in

=ERROR REPORT==== 28-Mar-2017::07:33:33 ===
** gen_event handler rabbit_mgmt_reset_handler crashed.
** Was installed in rabbit_event
** Last event was: {event,plugins_changed,
                          [{enabled,[prometheus,prometheus_rabbitmq_exporter]},
                           {disabled,[]}],
                          none,1490686413890}
** When handler state == []
** Reason == {'function not exported',
                 [{prometheus_http,setup,[],[]},
                  {prometheus_rabbitmq_exporter,dispatcher,0,
                      [{file,"src/prometheus_rabbitmq_exporter.erl"},
                       {line,11}]},
                  {rabbit_mgmt_dispatcher,'-build_dispatcher/1-lc$^2/1-2-',1,
                      [{file,"src/rabbit_mgmt_dispatcher.erl"},{line,32}]},
                  {rabbit_mgmt_dispatcher,'-build_dispatcher/1-lc$^2/1-2-',1,
                      [{file,"src/rabbit_mgmt_dispatcher.erl"},{line,33}]},
                  {rabbit_mgmt_dispatcher,build_dispatcher,1,
                      [{file,"src/rabbit_mgmt_dispatcher.erl"},{line,32}]},
                  {rabbit_mgmt_app,register_context,2,
                      [{file,"src/rabbit_mgmt_app.erl"},{line,50}]},
                  {rabbit_mgmt_reset_handler,handle_event,2,
                      [{file,"src/rabbit_mgmt_reset_handler.erl"},{line,55}]},
                  {gen_event,server_update,4,
                      [{file,"gen_event.erl"},{line,533}]}]}

and neither the WebGUI nor the prometheus endpoint are working any longer.

I guess some of the changes in RabbitMQ 3.6.8 are not compatible with your plugin :(

As for our situation in general: We are running multiple brokers, each with 10k to 100k queues & connections. This leads to two issues:

1. There is a significant RAM and CPU impact on the brokers (the resource consumption approx. triples and the brokers have a tendency to crash more easily) if we export the queue-related metrics. (Also running some queries like rabbitmqctl list_queues becomes quite slow and creates a significant impact on CPU/RAM.)
2. We generate quite a lot of data we don't want/need: Our prometheus becomes quite busy handling the data for every queue, while we are only interested in the total number of queues (and possibly the number of delivered messages). Right now we use the 'normal' api endpoint to gather this data, but it would be nice to use something more native.

Does this help to clarify the situation?
Thanks again for your great work and your help!

[deadtrickster]

Hi!

Setup & Configuration.

JSON encode error: {bad_term,{<<"x-priority">>,signedint,10}}
While encoding:

My plugin doesn't use JSON at all :-(. And before suggesting you this config snippet I've tested it on my fresh 3.6.8 install. But let's forget for the moment about this.

{enabled,[prometheus,prometheus_rabbitmq_exporter]}

New exporter version depends on two more plugins - accept and prometheus-httpd. Please download them from the release and try to enable it.

Btw I also pushed docker image too. Maybe you can use it as a base image. https://cloud.docker.com/swarm/deadtrickster/repository/registry-1.docker.io/deadtrickster/rabbitmq_prometheus/general.

Load handling

The quickest solution is to just don't use queues collector:

[
 {rabbit, [
           {loopback_users, []}
          ]},
 {prometheus,
  [{collectors,
    [
     %% Standard prometheus collectors
     prometheus_vm_statistics_collector,
     prometheus_vm_system_info_collector,
     prometheus_vm_memory_collector,
     prometheus_mnesia_collector,

     %% Process Info Collector
     prometheus_process_collector,

     %% Rabbitmq collectors
     prometheus_rabbitmq_overview_collector,
     prometheus_rabbitmq_exchanges_collector,
     prometheus_rabbitmq_mnesia_tables_collector,
     prometheus_rabbitmq_nodes_collector
    ]}
  ]}
].

This config will give you only queues count.

To filter queues by name we need to list them, right? So I'm curious what you are using now: Right now we use the 'normal' api endpoint to gather this data.

Thanks!

[Thubo]

On the "JSON-Error":

I'm starting the following image:

docker run --rm -p 15672:15672 -p 5672:5672 deadtrickster/rabbitmq_prometheus:3.6.8.1

Next I create an exchange/queue, send a message to the exchange and consume the queue using pika 0.10.0.

Navigating to the GUI results in:

The same message also appears in the logs:

=ERROR REPORT==== 9-Apr-2017::09:21:22 ===
JSON encode error: {bad_term,{<<"x-priority">>,signedint,10}}
While encoding:
[{incoming,[]},
 {deliveries,[]},
 {messages_details,
     [{avg,0.0},
      {avg_rate,0.0},
      {samples,
          [[{timestamp,1491729685000},{sample,0}],
           [{timestamp,1491729680000},{sample,0}],
[...]

---

As for the endpoint I mentioned: The management plugin provides the /api/overview endpoint.

The returned JSON holds (among other information):

  "message_stats": {
    "publish": 1,
    "publish_details": {
      "rate": 0.2
    },
    "confirm": 0,
    "confirm_details": {
      "rate": 0
    },
    "return_unroutable": 0,
    "return_unroutable_details": {
      "rate": 0
    },
    "disk_reads": 0,
    "disk_reads_details": {
      "rate": 0
    },
    "disk_writes": 0,
    "disk_writes_details": {
      "rate": 0
    },
    "get": 0,
    "get_details": {
      "rate": 0
    },
    "get_no_ack": 0,
    "get_no_ack_details": {
      "rate": 0
    },
    "deliver": 0,
    "deliver_details": {
      "rate": 0
    },
    "deliver_no_ack": 1,
    "deliver_no_ack_details": {
      "rate": 0.2
    },
    "redeliver": 0,
    "redeliver_details": {
      "rate": 0
    },
    "ack": 0,
    "ack_details": {
      "rate": 0
    },
    "deliver_get": 1,
    "deliver_get_details": {
      "rate": 0.2
    }
  },
  "queue_totals": {
    "messages_ready": 0,
    "messages_ready_details": {
      "rate": 0
    },
    "messages_unacknowledged": 0,
    "messages_unacknowledged_details": {
      "rate": 0
    },
    "messages": 0,
    "messages_details": {
      "rate": 0
    }
  },
  "object_totals": {
    "consumers": 1,
    "queues": 1,
    "exchanges": 9,
    "connections": 1,
    "channels":

It seems - I haven't done any in depth analysis so far (!) - this endpoint is cached or a least lot less expensive, since we can use it to collect the metrics we need without the mentioned performance impact.

Right now I cannot test the config example you posted - but I will report back to you as soon as I can.

Hope this helps.
Thanks!

[deadtrickster]

Hi,

Did you try suggested config yet? I will try to look at /api/overview shortly.
As for JSON... Can you please disable all prometheus stuff and see how it goes?

Thanks!

[deadtrickster]

/api/overview returns cluster wide data. Since prometheus scrapes each node I'm breaking down this cluster-wide stat to per-node stat.

rabbitmq_message_stats_deliver_get{node="nonode@host"} = 1

Thoughts?

[deadtrickster]

actually since all metrics are cluster-wide I don't think splitting by node just here makes sense