Consumer lag is climbing and never clearing
A consumer group that cannot keep up with the producer rate, often because of partition skew, a slow downstream, or a consumer that fetches too conservatively.
Lag draining to zero on steady state
Running Apache Kafka?
Kafka clusters usually come to us with one of three symptoms: consumer groups that cannot keep up, rebalance storms that make the whole consumer side flap, or a broker that has quietly become a single point of failure for the entire platform. We measure before we touch anything, and the fix is almost always tuning before it is scaling.
What it is
Apache Kafka is a distributed event streaming platform originally built at LinkedIn and open sourced in 2011. It is now the default backbone for event-driven systems: a publish/subscribe log with durable storage, partitioning for horizontal scale, consumer groups for parallel reads, and replication for durability.
A production Kafka deployment involves brokers that store partitions, ZooKeeper or KRaft for metadata, producers and consumers for traffic, and usually Schema Registry, Kafka Connect and a stream processor on top. Managed offerings include Confluent Cloud, AWS MSK and Aiven, each with slightly different operational characteristics and escape hatches.
The honest version
No technology is all upside. This is where Apache Kafka earns its place and where it tends to cause trouble.
Sound familiar?
The Apache Kafka problems that bring people to us, and what you have once each one is fixed.
A consumer group that cannot keep up with the producer rate, often because of partition skew, a slow downstream, or a consumer that fetches too conservatively.
Lag draining to zero on steady state
Session timeout set too low, max.poll.interval too short for the real work, or consumers that crash under load and trigger continuous group rebalances.
Group stable, no spurious rebalances
A broker under I/O pressure, a network link to a follower broker that is slow, or a replication quota set too low for the current traffic.
ISR fully caught up
A payload that has grown past the broker's max.message.bytes, often because of embedded images or large audit blobs, and the producer path has no guardrail.
Payloads reshaped or limits raised
A consumer that was behind long enough that retention trimmed the segment it was going to read next, and it does not know whether to restart from earliest or latest.
Reset strategy and retention aligned
A broker restart or network partition that triggers leader elections, and client code that does not handle the resulting metadata refresh gracefully.
Clients updated and tested for failover
Retention set long, partitions heavier than expected, and the broker is about to hit disk-full, which will take replication down with it.
Retention revised and partitions balanced
A topic auto-created with replication factor 3 but only 2 brokers available, so the create fails and producers start throwing.
Broker count and replication plan aligned
What we do
Hire us for one broken consumer group, or let us run the whole Kafka platform.
How it works
Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.
We fix the cause, not the symptom. If something is down, getting it back up comes first.
Backups, monitoring, updates and access, so the same problem can't quietly come back.
You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.
Often found together
We work on the whole stack, so the problem can be wherever it actually is.
Usually the processing rate is below the arrival rate on a steady state basis, not a transient spike. Look at per-partition lag first, because one hot partition with skewed keys can dominate the lag number. Then check whether the downstream (database, API, warehouse) is the real bottleneck, because adding consumers will not help if the next hop is already saturated.
Most rebalance storms trace to one of two things. Either max.poll.interval.ms is shorter than the real work the consumer does between polls, so the group coordinator thinks a healthy consumer is dead, or consumers are crashing on bad messages and taking the group through a cycle every time. Fix the config and add a dead-letter path for poison messages.
Confluent Cloud is the smoothest operationally but the most expensive. MSK is cheaper per gigabyte with more operational edges. Self-hosted is cheapest at steady state and expensive in on-call time, especially around upgrades and KRaft migration. We help model the total cost including engineering time before picking a direction.
Adopt Schema Registry early, agree on backward compatibility as the default, enforce it in CI so a non-compatible schema cannot be registered by accident, and version your consumer code against specific schema versions. Teams that skip this step eventually learn why the hard way in production.
Yes. KRaft is now the recommended mode and ZooKeeper is being removed from Kafka. The migration is supported in-place from recent Kafka versions but requires the cluster to be on a supported version and the controller plane to be upgraded first, so it is a planned exercise rather than a one-click change.
We give estimates for free. Describe the cluster, the symptoms and the scale and we respond with a price before any work starts.
Free estimate
Tell us the cluster shape and the symptom and we come back with a scoped plan and a price.
Your message is with a real person, and we'll get back to you soon.
Rather talk now? Call us: