Running Apache Kafka?

Kafka consumers lagging or brokers misbehaving? We fix it.

Kafka clusters usually come to us with one of three symptoms: consumer groups that cannot keep up, rebalance storms that make the whole consumer side flap, or a broker that has quietly become a single point of failure for the entire platform. We measure before we touch anything, and the fix is almost always tuning before it is scaling.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

kafka-broker.log

still broken

What it is

Apache Kafka at a glance

Apache Kafka is a distributed event streaming platform originally built at LinkedIn and open sourced in 2011. It is now the default backbone for event-driven systems: a publish/subscribe log with durable storage, partitioning for horizontal scale, consumer groups for parallel reads, and replication for durability.

A production Kafka deployment involves brokers that store partitions, ZooKeeper or KRaft for metadata, producers and consumers for traffic, and usually Schema Registry, Kafka Connect and a stream processor on top. Managed offerings include Confluent Cloud, AWS MSK and Aiven, each with slightly different operational characteristics and escape hatches.

Type
Distributed event streaming platform
First released
2011 (LinkedIn open source), 2012 Apache
License
Apache 2.0
Maintained by
Apache Software Foundation
Typically used for
Event streaming, async messaging, log aggregation, change-data-capture, stream processing

The honest version

Apache Kafka pros and cons

No technology is all upside. This is where Apache Kafka earns its place and where it tends to cause trouble.

Where Apache Kafka is strong

  • Durable, replayable logOnce events are in Kafka with sensible retention, new consumers can rebuild state from history rather than depending on a one-shot delivery guarantee.
  • Scales horizontally in both directionsAdding partitions increases parallelism on the consumer side, adding brokers increases storage and throughput, so the architecture does not need rewriting as load grows.
  • Strong ecosystem around itKafka Connect, Schema Registry, ksqlDB, Debezium, Flink and Spark Streaming all plug in, which means most of the hard problems already have proven integration paths.
  • Real exactly-once semanticsThe transactional producer and idempotent writes give end-to-end exactly-once within a Kafka workload, which is genuinely valuable for payments and billing event streams.
  • Managed options are matureConfluent Cloud, MSK and Aiven remove most of the broker, upgrade and ZooKeeper work that used to dominate running Kafka yourself.

Where Apache Kafka bites

  • Operating it yourself is genuinely hardPartitions, replication factors, ISR tuning, retention and disk management, broker JVMs, network throughput, and now KRaft migrations are all specialist topics. A naive self-managed cluster is a time bomb.
  • Rebalances hurt more than most teams expectA consumer group rebalance stops the whole group, and a misconfigured group that rebalances every few minutes will destroy throughput without obviously failing.
  • Partition count is a long-term commitmentYou can add partitions but not easily remove them, and the choice affects consumer parallelism, ordering guarantees and broker load forever after.
  • Schema evolution is not freeWithout Schema Registry and discipline, producers break consumers quietly, and compatibility rules (backward, forward, full) have to be agreed and enforced across teams.
  • Costs climb with retention and replicationThree-way replication times multi-day retention times high throughput is a lot of expensive disk, and managed services price that disk accordingly.
  • Observability has to be built inConsumer lag, under-replicated partitions, ISR shrink, and broker disk pressure are not visible without proper metrics and alerting, and by the time they are visible in the UI it is late.

Sound familiar?

Common Apache Kafka problems we fix

The Apache Kafka problems that bring people to us, and what you have once each one is fixed.

Consumer lag 8,412,007 messages

Consumer lag is climbing and never clearing

A consumer group that cannot keep up with the producer rate, often because of partition skew, a slow downstream, or a consumer that fetches too conservatively.

Lag draining to zero on steady state

rebalanced 14 times in 10 minutes

Consumer group is in a rebalance storm

Session timeout set too low, max.poll.interval too short for the real work, or consumers that crash under load and trigger continuous group rebalances.

Group stable, no spurious rebalances

Under-replicated partitions: 42

Replication is falling behind

A broker under I/O pressure, a network link to a follower broker that is slow, or a replication quota set too low for the current traffic.

ISR fully caught up

RecordTooLargeException 3MB

Producers are hitting message size limits

A payload that has grown past the broker's max.message.bytes, often because of embedded images or large audit blobs, and the producer path has no guardrail.

Payloads reshaped or limits raised

OffsetOutOfRangeException

Consumers fall off the end of the log

A consumer that was behind long enough that retention trimmed the segment it was going to read next, and it does not know whether to restart from earliest or latest.

Reset strategy and retention aligned

NotLeaderOrFollowerException

Leadership changes confuse clients

A broker restart or network partition that triggers leader elections, and client code that does not handle the resulting metadata refresh gracefully.

Clients updated and tested for failover

Broker 3 disk 92% full

A broker is running out of disk

Retention set long, partitions heavier than expected, and the broker is about to hit disk-full, which will take replication down with it.

Retention revised and partitions balanced

InvalidReplicationFactorException

Topic creation fails after a broker loss

A topic auto-created with replication factor 3 but only 2 brokers available, so the create fails and producers start throwing.

Broker count and replication plan aligned

What we do

Kafka help, from rescue to tuning

Hire us for one broken consumer group, or let us run the whole Kafka platform.

Rescue
Lag climbing, rebalances storming, or a broker about to fall off the edge. We stabilize the cluster first and then write up what caused it.
Consumer tuning
session.timeout.ms, max.poll.interval.ms, max.poll.records, fetch sizes and partition assignment strategies tuned to your actual workload rather than defaults.
Broker and cluster design
Replication factor, partition count, retention, disk sizing and KRaft migration planned against the traffic you expect, not the traffic you have today.
Schema Registry and contracts
Avro or Protobuf schemas with compatibility rules, enforced in CI so producers cannot break consumers at deploy time.
Managed migrations
Moving onto or between Confluent Cloud, AWS MSK and self-hosted Kafka, with mirrored traffic and a rollback path.
Managed Tech
An ongoing engagement that owns broker health, lag dashboards, upgrade planning and incident response when a partition decides to fall over on a Sunday.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Apache Kafka alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Apache Kafka questions

Something else? Call+1 737 727 7327

Why is our Kafka consumer lag not clearing?

Usually the processing rate is below the arrival rate on a steady state basis, not a transient spike. Look at per-partition lag first, because one hot partition with skewed keys can dominate the lag number. Then check whether the downstream (database, API, warehouse) is the real bottleneck, because adding consumers will not help if the next hop is already saturated.

Our consumer group keeps rebalancing, how do we stop it?

Most rebalance storms trace to one of two things. Either max.poll.interval.ms is shorter than the real work the consumer does between polls, so the group coordinator thinks a healthy consumer is dead, or consumers are crashing on bad messages and taking the group through a cycle every time. Fix the config and add a dead-letter path for poison messages.

Should we use Confluent Cloud, AWS MSK, or self-host Kafka?

Confluent Cloud is the smoothest operationally but the most expensive. MSK is cheaper per gigabyte with more operational edges. Self-hosted is cheapest at steady state and expensive in on-call time, especially around upgrades and KRaft migration. We help model the total cost including engineering time before picking a direction.

How do we handle Kafka schema evolution without breaking consumers?

Adopt Schema Registry early, agree on backward compatibility as the default, enforce it in CI so a non-compatible schema cannot be registered by accident, and version your consumer code against specific schema versions. Teams that skip this step eventually learn why the hard way in production.

Can you migrate our Kafka cluster from ZooKeeper to KRaft?

Yes. KRaft is now the recommended mode and ZooKeeper is being removed from Kafka. The migration is supported in-place from recent Kafka versions but requires the cluster to be on a supported version and the controller plane to be upgraded first, so it is a planned exercise rather than a one-click change.

How much does Kafka help cost?

We give estimates for free. Describe the cluster, the symptoms and the scale and we respond with a price before any work starts.

Free estimate

Kafka giving you trouble?

Tell us the cluster shape and the symptom and we come back with a scoped plan and a price.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.