Kafka Interview Questions

"A topic is a named stream of events, say orders. It isn't one big queue, though. It's split into partitions, and each partition is an append-only log: new records go on the end and nothing is changed in place. Every record in a partition gets an offset, which is just its position, counting up from zero. Offsets only mean something inside one partition, so offset 42 in partition 0 and offset 42 in partition 1 are different records. Partitions are how Kafka scales: they're spread across brokers, and different consumers can read different partitions in parallel. Reading doesn't delete anything. A consumer just remembers the offset it has reached, which is why you can replay a topic by moving back to an earlier offset."

"Kafka guarantees order only inside a partition. Across partitions of the same topic there's no global order at all. So if I need all events for one order to be processed in sequence, I give them the same key, the order ID, and they all land in the same partition. There are a few ways to break that by accident. On the producer side, if retries are on and several requests are in flight without idempotence, a failed batch can be retried after a later one succeeds, and the two swap places. The idempotent producer fixes that. Adding partitions to a topic changes which partition a key maps to, so old and new events for a key can end up in different places. And on the consumer side, if I hand records to a thread pool, I've thrown the ordering away myself."

"The key decides which partition a record goes to. The default partitioner hashes the key and maps it to one of the partitions, so every record for customer 17 lands in the same place as long as the partition count doesn't change. That gives me ordering per customer, and it keeps one entity's events together, which matters for stateful processing and for compacted topics, where Kafka keeps the latest value per key. If the key is null, the producer just spreads records across partitions. Newer clients use a sticky approach: they fill a batch for one partition before moving to another, rather than going strictly round robin, which gives bigger batches and fewer requests. That's fine for independent events like page views, but there's no ordering between them. The thing to watch with keys is skew: if one key is far busier than the rest, its partition becomes a hot spot."

"Kafka is a log, not a queue. Records stay for the retention period whether anyone reads them or not, consumers pull and track their own position, and any number of consumer groups can read the same data independently. That makes replay easy: a new service can read last week's events from the start. The trade-offs are that ordering and parallelism both come from partitions, and acknowledgement is by offset, so one stuck record holds up its partition. A traditional broker like RabbitMQ routes messages into queues and removes each one once a consumer acknowledges it. It gives you per-message acks, flexible routing and priorities, and dead-lettering and delayed retries are easy to set up. So I'd pick Kafka for high-volume event streams that several teams consume or need to replay, like order or click events. I'd pick a classic broker for distributing work, where each job should be done once by one worker and retried on its own."

"A broker is a Kafka server. It stores partition replicas on its disks, handles produce and fetch requests from clients, and copies data to and from other brokers for replication. A cluster also needs something that holds the metadata: which brokers are alive, which replica is the leader for each partition, and topic settings. That's the controller's job. For a long time that metadata lived in ZooKeeper, a separate system you had to run and secure alongside Kafka. Now Kafka uses KRaft mode, where a small quorum of controller nodes keeps the metadata in an internal log and agrees on it with a Raft-based protocol. It means one system to operate instead of two, faster recovery when the active controller fails, and room for far more partitions. From Kafka 4.0 on, KRaft is the only mode, so an older ZooKeeper-based cluster has to be migrated to KRaft on a 3.x release before it can upgrade."

"I start from the consumer side, because a partition is read by only one consumer in a group, so the partition count is the ceiling on parallelism. If the topic needs to handle a certain rate and one consumer instance can process about a quarter of it, I need at least four partitions, and I'd add headroom for growth. I also check the producer side and how many brokers the load should spread across. I don't go wildly high, because every partition adds open files, replication traffic, metadata, and time during leader elections, and batches get smaller. The reason to get it right early is that you can add partitions but you can't remove them, and adding them changes which partition each key hashes to. So for a keyed topic, a customer's new events start landing somewhere other than its old ones, which breaks per-key ordering across the change. For keyed topics I size generously up front, or create a new topic and migrate."

"Without it, retries can create duplicates. The producer sends a batch, the leader writes it, but the acknowledgement gets lost in a network blip. The producer times out, retries, and now the record is in the log twice. With idempotence on, the producer gets a producer ID from the broker, and every batch it sends to a partition carries a sequence number. The broker remembers the last sequence it accepted from that producer for each partition, so when a retry arrives with a number it has already written, it acknowledges it without writing it again. It also rejects out-of-order sequences, which is why idempotence keeps ordering safe with several requests in flight. It needs acks set to all, and it's on by default in recent clients. The limit is scope: if my application crashes and resends the same order event on restart, that's a brand-new record as far as Kafka knows. That needs transactions or an idempotent consumer."

"First I'd check the code, because the most common cause is calling get on every send, which turns an asynchronous producer into a synchronous one that waits a round trip per record. After that I'd look at batching. The producer groups records per partition, and linger.ms lets it wait a few milliseconds so batches fill up, while batch.size caps how big each one gets. Bigger batches mean far fewer requests, at the cost of a little latency. Then compression: lz4 or zstd cuts network and disk use a lot for text formats like JSON, costing some CPU, and it compresses better when batches are bigger. If sends are blocking because the buffer is full, buffer.memory matters, but that's usually a sign the brokers can't keep up. I wouldn't drop acks from all to get speed unless the business has agreed it can lose data. And if one partition is overloaded, more partitions spread the work across brokers."

"I set the key to the customer ID, so the default partitioner sends every event for one customer to the same partition, and they stay in order. I use acks=all with idempotence on, so retries after a network blip don't lose, duplicate or reorder anything. The important part is that send doesn't send anything right away. It adds the record to a buffer and returns a future. To know whether it worked, I pass a callback. When the broker acknowledges, I get the partition and offset where it was stored. If the exception isn't null, the producer has already given up retrying, so I have to act on it: log it with the order ID, alert, or write it somewhere I can resend from. Blocking on get for each record also works, but it kills throughput. Finally, flush or close before the program exits, or anything still in the buffer is lost."

"With acks=0 the producer doesn't wait for anything, so it's fast but data can vanish silently. With acks=1 the leader writes the record to its own log and replies, but if the leader dies before the followers copy it, that record is gone even though the producer was told it succeeded. With acks=all the leader waits until every replica in the in-sync set has it. The catch is that the in-sync set can shrink. If two followers fall behind, it's just the leader, and all means one copy. That's what min.insync.replicas fixes. It sets the minimum in-sync count for an acks=all write to succeed. Below that, the producer gets a not-enough-replicas error and retries, so the topic stops accepting writes instead of pretending they're safe. The usual durable setup is replication factor three, min in-sync two, acks=all. One broker can be down and writes still work, and every acknowledged write is on at least two brokers."

"Each partition has a replication factor, say three, so there are three copies on three brokers. One copy is the leader, and producers and consumers normally talk to it. The followers keep fetching from the leader to stay current. The ISR, the in-sync replica set, is the leader plus every follower that has kept up within a time limit. If a follower is slow or down for too long it's dropped from the ISR, and it rejoins once it has caught up again. Consumers only see records that every in-sync replica has, which is the high watermark. When the leader's broker dies, the controller notices and promotes one of the ISR members to leader. Because the ISR had every acknowledged record, nothing written with acks=all is lost. Clients get an error, refresh their metadata, and carry on with the new leader. If no in-sync replica is left, the partition stays offline unless unclean leader election is turned on, which trades data loss for availability."

"Consumers that share a group ID act as one logical subscriber. Kafka splits the topic's partitions among them, and each partition is owned by exactly one consumer in the group at a time, though one consumer can own several. That's how a group processes a topic in parallel while each record is handled once per group. Different groups are completely independent. If billing and analytics both read the orders topic, each group gets every record and keeps its own offsets, so one falling behind doesn't affect the other. If a group has more consumers than partitions, say six consumers on a four-partition topic, two of them get nothing and sit idle. They're not useless, since they take over if another consumer dies, but they add no throughput. So if I need more parallelism, adding consumers only helps up to the partition count."

"A rebalance happens whenever group membership or the partition set changes. A consumer joins during a scale-up or deploy, one leaves cleanly, one stops heartbeating and passes the session timeout, or one takes longer than max.poll.interval.ms between polls and is kicked out even though it's alive. Adding partitions to the topic triggers one too. It hurts because with the classic eager protocol every consumer gives up all its partitions and the whole group stops until assignments are handed out again. Anything processed but not committed gets done a second time. To soften it, I use the cooperative sticky assignor, so only partitions that actually move are revoked and everyone else keeps working. Static membership, setting group.instance.id, lets a pod restart within the session timeout and get its partitions back with no rebalance, which helps rolling deploys. I keep max.poll.records small enough that a batch always finishes well inside the poll interval, and I commit offsets in the revoke callback."

"Committed offsets live in an internal compacted topic called __consumer_offsets, keyed by group, topic and partition. The number stored is the next offset to read, not the last one processed, which trips people up. With auto-commit on, the client commits in the background on a timer, inside poll, using the offsets that earlier polls returned. If I process records synchronously in the poll loop, that works out as at-least-once. But if I hand records to another thread, a commit can go through before the work is done, and a crash then loses those records. And a crash just before a commit means reprocessing. With auto-commit off, I decide. I process the batch, then call commitSync, which blocks and retries, or commitAsync, which is faster but doesn't retry. A common pattern is commitAsync in the loop and one commitSync on shutdown. I commit per batch, not per record, because committing every record is slow."

"It tells a consumer where to start when its group has no valid committed offset for a partition. That's a brand-new group, or a group whose committed offset points at data that retention has already deleted. Earliest means start from the oldest record still kept, latest means start from the end and only see new records, and none means throw an error so I decide myself. Latest is the default in the Java client. The key thing is it doesn't apply on a normal restart. If the group has a valid committed offset, it resumes from there no matter what this says. The traps come from that. A new group on latest quietly skips everything already in the topic. And a group that's been stopped longer than the retention period comes back to an out-of-range offset and jumps, either skipping data or reprocessing a lot, depending on the setting. For anything important I'd rather use none and handle it on purpose."

"I switch off auto-commit so nothing is committed behind my back. In the loop I poll, process every record in the batch, and only then call commitSync. With no arguments, commitSync commits the positions from the last poll, which is the next offset after the last record of each partition. Since the commit comes after the work, a record is never marked done before it's handled, so nothing is lost. The flip side is duplicates. If the service crashes after handling half the batch, the next owner of those partitions starts again from the last commit and redoes those records. So the handler has to be idempotent, for example an upsert keyed on the order ID, or a check against processed event IDs. I commit per batch rather than per record, because per-record commits add a round trip each time. To stop cleanly, I'd call wakeup from another thread and close the consumer, which also leaves the group."

"On the consumer side it comes down to when I commit compared with when I process. At-most-once is commit first, then process. If I crash in between, that record is never handled, so it can be lost but never duplicated. That suits things like metrics where a gap is fine. At-least-once is process first, then commit. If I crash in between, the record comes back after restart, so nothing is lost but I can see duplicates. That's what most systems use, with idempotent handling. Exactly-once means each record's effect happens once. If I'm reading from Kafka and writing results back to Kafka, transactions give me that, because the output and the offset commit succeed or fail together. If the effect is in a database or an API call, Kafka can't cover it, so I get it with at-least-once plus an idempotent write, or by saving the offset in the same database transaction as the result."

"It's built from a few pieces. The producer is idempotent, so retries don't duplicate. On top of that it gets a transactional ID and uses transactions. For each batch I begin a transaction, send the output records, add the consumer's offsets to the same transaction with sendOffsetsToTransaction, then commit. The outputs and the offset commit either all happen or none do, so a crash never leaves output written without the input marked done, or the other way round. Downstream consumers set isolation.level to read_committed, so they skip aborted records and don't see a transaction until it commits. The transactional ID also fences zombies: if an old instance comes back after a restart, its epoch is out of date and its writes are rejected. Where it stops is anything outside Kafka. If my step calls a payment API or writes to a database, a retry can still repeat that, so those need idempotency keys or an outbox. And it costs some latency and throughput."

"At my last company, the loyalty team said they needed exactly-once delivery, because points must never be awarded twice. I asked where the effect actually happened, and it was a points ledger in a relational database, not a Kafka topic. So Kafka transactions wouldn't have helped: they cover writes to Kafka, not to their database. What they really needed was for each purchase event to change the balance once. We kept a normal at-least-once consumer and added a table of processed event IDs with a unique constraint, written in the same database transaction as the balance update. A duplicate insert fails, so a replayed event does nothing. To convince them, I killed a consumer halfway through a batch in staging, let it replay, and showed the balances were still right. It was simpler than transactions and added no latency. The lesson I took is to always ask what exactly-once means to the person asking. It usually means the effect must happen once."

"Retention is set per topic and has nothing to do with whether anyone has read the data. With the delete cleanup policy, there are two limits. retention.ms removes data once it's older than a set age, and retention.bytes caps how big each partition can get. Whichever is hit first wins. Under the hood each partition is a series of segment files, and Kafka deletes whole segments, never single messages. A segment only becomes eligible once it's closed and its newest record is past the retention time. The active segment, the one being written, is never deleted. So data can live somewhat longer than the setting, especially on quiet topics where a segment stays open a long time. One gotcha with retention.bytes is that it's per partition, so the topic's real disk use is roughly that limit times the number of partitions times the replication factor."

"With compaction, Kafka keeps at least the latest value for each key and removes older records with the same key in the background. So a compacted topic works like a table of current state. I'd use it for things like customer profiles by customer ID, or a changelog a service reads on startup to rebuild its cache. A new consumer can read from the start and end up with the current value for every key without replaying years of history, which time-based retention can't give you without deleting keys that haven't changed recently. To delete a key, you write a tombstone, the key with a null value. Compaction then removes the older values, and the tombstone itself after a configured delay, so consumers get a chance to see the delete. A few caveats: the active segment isn't compacted, so a consumer can still see several values per key, offsets keep their numbers so gaps appear, and every record must have a key."

"Kafka brokers just store bytes. They don't know or check the format, so if a producer renames a field, consumers find out at runtime when they break. A schema registry is a separate service that stores versioned schemas, in Avro, Protobuf or JSON Schema, usually one subject per topic. The producer's serializer registers or looks up the schema and puts a small schema ID at the front of each message, and the consumer's deserializer fetches the schema by that ID. The registry's real value is the compatibility check on every new version. Backward means the new schema can read data written with the old one, so you upgrade consumers first. Forward means old schemas can read data written with the new one, so producers can go first. Full means both. In practice, adding an optional field with a default is safe, while renaming a field or changing its type usually isn't, and the registry rejects it before it ever reaches the topic."

"At my last company I owned a shipments topic that four other teams read, and we needed to split a single address string into structured fields. Changing that field in place would have broken everyone at once. So I added the new structured fields as optional fields with defaults, alongside the old string, which the schema registry accepted as compatible. For a few weeks the producer filled in both. I posted the plan with dates in each team's channel and kept a small tracker of which consumer had switched, checking with each team rather than assuming. One team needed longer because of a release freeze, so we moved the date instead of forcing it. Once everyone had switched, we removed the old field in a new schema version. Nothing broke and nobody had to deploy in a rush. What I'd repeat is treating the event like a public API: never break it in place, and make the migration visible so people can plan around it."

"The order service owns an orders topic and publishes an OrderPlaced event keyed by order ID, so everything about one order stays in sequence. Payment, inventory and notifications each run their own consumer group, so they read independently and one slow service doesn't hold up the others. The first trap is the dual write. Saving the order to the database and publishing to Kafka are two separate systems, so a crash between them loses an event or announces an order that doesn't exist. I'd use the outbox pattern: write the event to an outbox table in the same database transaction, and have a relay or change data capture publish it. Every consumer is at-least-once, so each one dedupes by event ID. Records that will never succeed go to a dead letter topic with the error attached. Payment and inventory publish their own result events, and if payment fails after stock was reserved, inventory hears it and releases the stock. No distributed transaction needed."

"Offsets move forward in order, so if I keep retrying one bad record, everything behind it in that partition waits. The plain consumer has no built-in dead letter queue, so it's my call. First I sort the failure. If it's transient, like the database being down, I retry with backoff, and if it goes on I pause the partition rather than skip data. If it's permanent, like a malformed payload or a missing required field, no retry will fix it. Then I publish the raw record to a dead letter topic with headers for the error, the source topic, partition and offset, commit, and move on. I alert on the dead letter topic so it doesn't become a silent bin, and keep a small tool to replay records once the bug is fixed. I also catch deserialization failures, because those blow up before my handler even runs. If delayed retries are needed, retry topics work, but they break strict ordering for that key."

"Probably not, but I'd ask questions before saying so. What they've described is a job queue: each job done once, retried on its own, sometimes delayed. A classic queue gives all of that out of the box. On Kafka, a failing job blocks the ones behind it in its partition unless they build retry topics, there's no native delay, and parallelism is capped by partitions. For a few hundred jobs a day, that's a lot of machinery, and running Kafka, even a managed one, adds cost and a learning curve. The answer changes if the company already runs Kafka well with a platform team behind it, or if these jobs are really events other teams want to read or replay. So I'd ask what they need. Replay and several readers point to Kafka. One worker per job with retries points to a simple queue, or even a jobs table in the database they already have. Either way, I'd write the decision down."

"Lag is how far a consumer group is behind the newest data. For each partition, it's the log end offset minus the group's committed offset. I watch it per partition, not only as a total, because a total can hide one stuck partition. I also like lag measured as time, meaning how old the oldest unprocessed record is, because ten thousand messages behind means very different things on a quiet topic and a busy one. For a quick look, the consumer groups command shows current offset, log end offset and lag for every partition. For real monitoring I export it to dashboards and alert on lag that keeps growing, not on a single spike. The usual causes are slow processing, often a slow database or API call downstream, a traffic spike, too few consumers or partitions, one hot partition, frequent rebalances, or a consumer stuck retrying one bad record."

"Adding consumers can't help, because only one consumer in the group reads that partition. So I'd look at two things. First, is the consumer's offset moving at all? If it's frozen, it's probably stuck on one record or a hung downstream call, and the fix is to get that record to the dead letter topic and see why it hung. If it's moving but slowly, I compare incoming rates per partition. Usually one partition is getting far more data because of a hot key, like one huge customer, or lots of records sharing a default key such as unknown or an empty string. Short term, I can speed up that consumer by processing in parallel inside it, using a worker per key so per-key order still holds. Long term, the key needs fixing: a finer key if the business only needs ordering per sub-entity, or a separate topic for the big tenant. I'd also check the broker leading that partition isn't struggling."

"First I stop the bleeding by rolling back the producer release or pausing it. Then I work out the blast radius: which partitions and which offset range, which I can find from the release time, since consumers can look up offsets by timestamp. I tell all five teams straight away with those exact ranges, not a vague heads-up. Kafka doesn't let me edit or delete records in the middle of a log, so recovery happens on the consumer side. Each team chooses. Some can reset their group's offsets past the bad range, with the group stopped and a dry run first. Others let their dead letter handling catch the records. If the data matters, we publish corrected events with the same keys and make it clear they replace the bad ones. Afterwards I'd run a blameless postmortem and fix the gap that let it through: schema validation at write time through the registry, contract tests in CI, and a canary release for producers on shared topics."

"At my last company, the billing consumer's lag climbed every afternoon, and the group rebalanced over and over. At first it looked like a capacity problem, so someone added consumers, and it got worse. I read the consumer logs and saw members leaving the group because the time between polls had passed max.poll.interval.ms. Each record called a payment API that slowed down under afternoon load, and with a few hundred records per poll, some batches ran past the limit. The consumer got kicked out, its uncommitted batch went to someone else, who then hit the same slow API. It was a loop. The quick fix was lowering max.poll.records so every batch finished well inside the limit. The real fix was a proper timeout on the API call and an idempotency key, so a redone batch couldn't charge anyone twice. We added alerts on rebalance rate and on lag in time. What I took from it is that a rebalance storm is often slow processing in disguise."