Running RabbitMQ?

RabbitMQ queue backlog or broker struggling? We clear the backlog.

RabbitMQ is often fine for years and then suddenly isn't: a queue that nobody is draining, a channel.close 406 on every publisher after someone changed a flag, a quorum queue that will not elect a leader after a rolling restart, or a memory alarm that paused the whole broker. We work through the management UI, the publisher code and the policy stack until the backlog is gone and the next incident is boring.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

rabbit@broker.log

still failing

What it is

RabbitMQ at a glance

RabbitMQ is a message broker implementing AMQP 0-9-1 and, with plugins, MQTT, STOMP and newer AMQP revisions. A producer publishes to an exchange, the exchange routes to zero or more queues by binding rules, and consumers pull or are pushed messages off those queues with configurable acknowledgement semantics. It has been the default pick for task queues, event buses and inter-service messaging in many stacks for over a decade.

Modern RabbitMQ leans heavily on quorum queues, which use the Raft consensus algorithm for replication rather than the older mirrored classic queues that are now deprecated. Streams, added in 3.9, cover log-style use cases where Kafka would otherwise be the fit. The broker is now stewarded by Broadcom after the VMware acquisition, and the open-source release cadence has shifted compared to the earlier Pivotal years.

Type
AMQP message broker with support for streams, MQTT and STOMP
First released
2007 initial release
License
Mozilla Public License 2.0
Maintained by
Broadcom (VMware Tanzu) and the open-source community
Typically used for
Task queues, event-driven architectures, inter-service messaging, webhook dispatch, work distribution

The honest version

RabbitMQ pros and cons

No technology is all upside. This is where RabbitMQ earns its place and where it tends to cause trouble.

Where RabbitMQ is strong

  • Battle-tested routing semanticsDirect, topic, fanout and headers exchanges cover most routing scenarios cleanly, and the combination of exchanges, queues and bindings is flexible without being surprising.
  • Per-message acknowledgementWorkers explicitly ack or nack messages, and redelivery on crash is predictable, which is still an edge over systems that assume at-most-once by default.
  • Quorum queues add real durabilityRaft-replicated quorum queues give you a consistent, leader-elected queue that survives node failure without the surprises classic mirrored queues used to produce.
  • Management plugin is genuinely usefulThe HTTP API and web UI make it possible to see what is in a queue, who is bound to which exchange, and where the backlog is sitting, which saves hours of guesswork during incidents.
  • Wide client supportMature libraries exist for Python, Java, .NET, Node, Go, Ruby and Erlang itself. A small team can wire a producer and a consumer in under an afternoon.

Where RabbitMQ bites

  • It is an Erlang applicationOperating RabbitMQ at scale eventually requires understanding how Erlang schedulers, Mnesia and distribution ports behave. That is a different skill set than most backend teams are hiring for.
  • Classic mirrored queues are deprecatedOlder clusters relying on them need to be migrated to quorum queues or streams, and that migration is not a simple toggle because the operational characteristics are different.
  • Memory and disk alarms pause everythingWhen a resource alarm is tripped, the broker throttles publishers aggressively. That protects durability but can look like a full outage to applications that were not designed to handle backpressure.
  • Policy stack is powerful and confusingQueue arguments, policies and operator policies overlap, and a small misconfiguration can produce the channel.close 406 PRECONDITION_FAILED errors every operator sees eventually.
  • Not the right pick for every stream workloadFor append-only event logs with very high fan-out and long retention, Kafka or Redpanda are usually a better match. Streams in RabbitMQ help but do not fully close that gap.
  • Ownership drift post-acquisitionThe move under Broadcom has changed release cadence and commercial support terms in ways that enterprise users should look at carefully before standardising on the latest version.

Sound familiar?

Common RabbitMQ problems we fix

The RabbitMQ problems that bring people to us, and what you have once each one is fixed.

PRECONDITION_FAILED - inequivalent arg 'durable'

Publishers are being kicked off with 406 errors

A queue declared with different arguments from what the publisher expects, usually because one service was updated with new flags and another was not, so the second service's channel closes on declare.

Queue declarations aligned across services

queue has 1,243,502 messages ready

A queue is backed up and nobody is draining it

A consumer that crashed and did not restart, a prefetch and concurrency setting that caps throughput well below publish rate, or a dead-letter queue that was never meant to hold real traffic.

Backlog drained and prevented

memory resource limit alarm set

Memory alarm has paused publishers

A broker whose working set grew past the default 40 percent memory high watermark, often because a classic queue is holding unacknowledged messages that consumers never confirmed.

Memory pressure removed, publishers resumed

quorum queue leader election took 42s

Quorum queues are slow to recover

A rolling restart or network partition where Raft leader election takes seconds to tens of seconds, during which publishes to that queue fail or buffer upstream.

Quorum elections fast and predictable

disk_free_limit alarm: 47 MB free

Disk space triggers a broker-wide alarm

A node running close to the disk watermark because of a persistent queue that grew unbounded, with no disk-space headroom left to absorb a traffic spike.

Storage sized to real retention

ACCESS_REFUSED access to vhost

Users cannot connect after a cluster join

A vhost permissions table that was never replicated to a newly joined node, or a user whose tags were changed without updating the policy set it depends on.

Permissions and vhosts consistent

basic.nack requeue=true

Messages loop forever between consumer and queue

A consumer nacking a poison message with requeue=true, which puts it right back on the queue to be delivered again to the same broken handler, producing an endless loop.

Dead-letter path in place for poison messages

connection_closed_abruptly handshake_timeout

Clients disconnect during handshake

A heartbeat or TLS negotiation failing because of a load balancer with an aggressive idle timeout, or a client version that does not agree with the broker on the handshake frame.

Handshakes stable, connections durable

What we do

RabbitMQ help, from rescue to steady state

Call us for one dying consumer, or hand us the whole broker.

Rescue
A memory alarm pausing publishers, a backlog that will not drain, or a cluster node that will not rejoin. We stabilise the broker first and explain what caused it after.
Queue design
Classic, quorum or stream chosen deliberately per use case, with policies and arguments that match the durability and throughput you actually need.
Consumer reliability
Prefetch, concurrency, acknowledgement semantics and dead-letter paths reviewed against real workload, so a single bad message does not take an entire consumer fleet down.
Migration from mirrored queues
Moving deprecated mirrored classic queues onto quorum queues or streams with a plan for cutover, idempotency and rollback.
Upgrades and clustering
Minor-version upgrades rehearsed on a copy, Erlang version alignment, cluster partitions and the Mnesia bits that production teams usually hit the hard way.
Managed Tech
Backlog alerting, resource watermark tuning, policy reviews, upgrade planning, and an engineer on call when the broker throttles during a traffic spike.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running RabbitMQ alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

RabbitMQ questions

Something else? Call+1 737 727 7327

Should we use RabbitMQ or Kafka?

They solve overlapping problems but are shaped differently. RabbitMQ is excellent at per-message routing, work distribution and request-reply patterns. Kafka is better when you need an append-only log, long retention, and high fan-out of the same events to many consumers. For a typical task queue or integration bus, RabbitMQ is usually simpler to operate; for event streaming at scale, Kafka tends to win.

Why do we get PRECONDITION_FAILED errors from producers?

Almost always because a queue already exists with different arguments than the producer is trying to declare: durability flag, auto-delete, x-queue-type, dead-letter configuration. One service was updated, another was not, and their declarations now disagree. The fix is either aligning the declarations or using policies rather than per-queue arguments so one owner controls the definition.

How do we migrate from mirrored classic queues to quorum queues?

With a plan that treats them as different queues rather than a toggle. Typically we create the new quorum queue alongside, point new publishers at it, drain the old classic queue with a shovel or a consumer, and only then retire the old one. Idempotency on the consumer side is non-negotiable, because the migration will replay some messages.

Why did publishers get blocked when memory went up?

RabbitMQ sets memory and disk resource alarms that intentionally pause publishers when the node is at risk, to protect durability. That is a feature, not a bug, but it looks like an outage to applications that do not handle backpressure. The right response is usually a combination of lowering the working set, raising the memory ceiling, and making publishers tolerant of pause signals.

Can RabbitMQ run in a Kubernetes cluster?

Yes, with the official RabbitMQ Cluster Operator or Topology Operator. For most teams this is now the preferred way to run it, because the operator handles Erlang distribution networking, peer discovery and rolling upgrades. There are still gotchas around persistent volumes and pod identity that need deliberate thought.

How much does RabbitMQ help cost?

Estimates are complimentary. Describe the broker, its version, and the specific behaviour that is misbehaving, and you get a quote back before any config change lands.

Free estimate

RabbitMQ giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.