Kafka Interview Cheatsheet: Quick Revision Before Your Interview

Heading into a Kafka interview? This cheatsheet distills the key concepts, configs, and gotchas from every major Kafka topic — architecture, brokers, partitions, topics, producers, consumers, scaling, and KRaft — into one quick-revision guide. No fluff, just the things that actually come up in interviews.

1. Architecture & Core Concepts

What Kafka actually is

Kafka is a distributed, append-only commit log — not a traditional message queue. Messages are retained for a configurable window (e.g. 7 days) regardless of whether they've been consumed. Multiple independent consumer groups can read the same data at their own pace.

Two tiers

• Server tier: Brokers (storage) + Kafka Connect workers (source/sink connectors).
• Client tier: Producers, Consumers, Kafka Streams apps.

Bootstrap servers — a common interview trap

The bootstrap.servers list is only used for the initial metadata handshake. After one broker responds with the full cluster topology (broker addresses, partition leaders, ISRs), the client connects directly to partition leaders. The bootstrap broker is never a data proxy. 2–3 entries are enough for HA — you don't need to list every broker.

The advertised.listeners gotcha

If brokers advertise internal hostnames (e.g. Docker container names) but clients are external, the metadata handshake succeeds but direct connections to partition leaders will timeout. Always ensure advertised.listeners exposes addresses routable from the client network.

Multi-cluster topologies

• Active-Passive: One active cluster, one standby DR cluster. Data replicated via MirrorMaker 2 (MM2).
• Active-Active: Both clusters accept writes. Requires topic prefixing (e.g. us.orders, eu.orders) to prevent infinite replication loops.
• Hub-and-Spoke: Edge clusters send data to a central hub.
• Stretched (Metro): Single logical cluster across nearby DCs (<10ms RTT). RPO = 0.

2. Kafka Broker Internals

What a broker does

1. Accepts writes from producers (appends to segment log files).
2. Persists data to disk using sequential I/O (no random seeks).
3. Serves fetch requests from consumers and follower replicas.

Partition Leadership

• Every partition has one Leader (handles all writes) and N-1 Followers (passive replicas).
• Kafka distributes leadership evenly across brokers so no single server is a bottleneck.
• Leader and follower replicas are never on the same physical host — that would defeat replication.

Storage: Segments + Sparse Indexes

• Each partition directory has rolling segment files (default 1 GB each).
• .log — raw append-only record batches.
• .index — sparse memory-mapped offset → byte-position lookup (binary search).
• .timeindex — timestamp → offset lookup (enables offsetsForTimes()).

OS-level speed tricks

• OS Page Cache: Kafka avoids JVM heap buffering entirely. Recent writes are served directly from kernel RAM, bypassing GC pauses.
• Zero-Copy (sendfile): Data transfers from page cache to NIC via DMA without ever touching user space. Halves CPU cycles and context switches vs. standard Java I/O.

Broker concurrency model

Brokers use a Reactor pattern: Acceptor threads → NIO Network threads → Request Queue → I/O Handler threads → Response Queue. This decouples connection management from disk I/O, allowing tens of thousands of concurrent connections.

3. Kafka Partitions

What partitions give you

• Horizontal scale: Distributes storage and I/O across brokers.
• Ordering guarantee: Strict chronological order within a single partition (never across partitions).
• Parallelism ceiling: The number of partitions is the hard maximum for consumer group concurrency.

Producer routing (on the client side, not the broker)

1. Explicit partition set → direct route.
2. Key provided → MurmurHash2(key) % numPartitions (deterministic; same key always → same partition).
3. No key → Sticky Partitioner: fills a batch in one partition before rotating. Better compression and fewer TCP frames than round-robin.

LEO vs. High Watermark

• Log End Offset (LEO): Next offset to be written. Advances immediately on leader append — not yet durable.
• High Watermark (HW): Highest offset replicated to ALL ISR members. Consumers can never read past the HW. This prevents reading “ghost messages” that disappear on leader crash.

ISR (In-Sync Replicas)

A follower stays in the ISR if it continuously fetches within replica.lag.time.max.ms (default 30s). If it falls behind, the leader evicts it. When the ISR shrinks to 1 (just the leader) and min.insync.replicas=1, acks=all silently becomes equivalent to acks=1 — a data loss risk.

Durability recipe (production gold standard)

replication.factor = 3
min.insync.replicas = 2
acks = all
enable.idempotence = true

This tolerates one broker failure while safely pausing ingestion if a second broker fails.

4. Kafka Topics

Topic vs. queue

Unlike RabbitMQ/JMS where messages are removed after consumption, Kafka topics are append-only logs retained by time/size. Multiple independent consumer groups read the same topic concurrently without interference.

Partition sizing formula

Partitions = max(
  Target produce throughput (MB/s) / Single partition write limit,
  Target consume throughput (MB/s) / Single consumer read speed,
  Target message rate (msgs/s) / Single consumer processing speed
)

Key insight: The real bottleneck is almost always consumer processing speed (DB writes, API calls), not raw MB/s. A 100-byte-record topic at 100K msgs/s may need 100 partitions even though the data volume is only 10 MB/s.

Retention policies

• cleanup.policy=delete: Delete segments older than retention.ms or exceeding retention.bytes. Default for event streams.
• cleanup.policy=compact: Retain only the latest record per key. Used for state changelogs, CDC, user profiles. Deleting a key requires a tombstone (record with null value); tombstones are kept for delete.retention.ms (default 24h) so consumers can read the deletion event.

Dead Letter Queue (DLQ) pattern

1. Main topic: orders.events.v1
2. Retry topic: orders.events.retry.v1 (transient failures, exponential backoff)
3. DLQ: orders.events.dlq.v1 (permanent failures for human inspection)

Naming convention

<env>.<domain>.<entity>.<event_type>.<version> — e.g. prod.billing.invoices.created.v1

5. Kafka Producer

Two-thread architecture

• Application thread: Calls send(), serializes records, routes via partitioner, writes to RecordAccumulator buffer.
• Sender I/O thread: Background thread that drains batches and flushes them to broker leaders via TCP.

The app thread never blocks on network I/O. It only writes to an in-memory buffer and returns a Future<RecordMetadata>.

Key tuning configs

acks levels

• acks=0: Fire-and-forget. No durability guarantee. Use for metrics/clickstream.
• acks=1: Leader writes to disk. Un-replicated records lost if leader crashes. Use for standard log pipelines.
• acks=all (or -1): Full ISR quorum must acknowledge. Zero data loss when combined with min.insync.replicas=2. Use for financial data.

Idempotence — how duplicates are prevented

With enable.idempotence=true (default since Kafka 3.0):

1. Producer gets a unique PID (Producer ID) from the cluster on startup.
2. Each batch gets a monotonically incrementing sequence number per partition.
3. Broker caches the last 5 sequence numbers per PID per partition. Duplicate sequences are silently discarded and re-ACKed.

BufferPool — prevents GC pauses

The producer pre-allocates a pool of ByteBuffer instances sized to batch.size. After a broker ACK, buffers are returned to the pool — not garbage collected. This eliminates Stop-the-World GC pauses in high-throughput pipelines.

6. Kafka Consumer

Pull model vs. push

Kafka consumers pull data at their own rate. This gives built-in backpressure (if your DB is slow, just don't call poll() until ready), efficient batch fetching, and free replay by rewinding offsets.

Consumer groups

• Each partition assigned to exactly one consumer within the group at any time.
• If consumers > partitions: extra consumers sit idle as hot standbys.
• The Group Coordinator (a broker) handles partition distribution, offset tracking (__consumer_offsets), and failover.

Two critical timeouts

Rebalancing: Eager vs. Cooperative Sticky

• Eager (old): All consumers drop all partitions, stop, rejoin. Full “stop-the-world” pause.
• CooperativeSticky (modern): Only the partitions that need to migrate are revoked. Unaffected partitions keep processing uninterrupted. Set via CooperativeStickyAssignor.

Offset commit strategies

• Auto-commit: Simple but dangerous — commits on a timer regardless of whether processing succeeded. Risk of both data loss and duplicates.
• Manual sync (commitSync()): Commits only after business logic completes. Blocks thread, lower throughput.
• Async in loop + sync on shutdown: Best of both. commitAsync() on hot path for throughput, commitSync() in shutdown hook for safety.

Consumer lag

Consumer Lag = High Watermark (HW) - Last Committed Offset

Growing lag = consumers can't keep up. Primary signal to scale up (more partitions + more instances).

7. Consumer Scaling

The 1:1 partition rule

Maximum Active Consumers = Total Topic Partitions. Consumers beyond that number sit idle. Plan partition counts for your peak spike, not average load. If your baseline needs 3 threads but spikes need 30, provision 30+ partitions at topic creation — you cannot safely reduce partitions later.

Spring Kafka: one thread = one KafkaConsumer instance

The Java KafkaConsumer is strictly single-threaded. Spring's concurrency="N" instantiates N separate consumer instances, each with a dedicated thread running an independent poll() loop.

Deployment: monolith vs. dedicated microservices

• Monolith: All consumers in one JVM. Simple, but an OOM in the analytics consumer kills your payment pipeline. Scaling one consumer scales them all.
• Dedicated microservices: Isolated fault domains. Scale each topic consumer independently. Preferred for mission-critical pipelines.

KEDA autoscaling (scale on lag, not CPU)

Traditional HPA scales on CPU — useless for Kafka consumers that sit at 15% CPU while lag builds to millions. KEDA queries broker lag metrics and scales pods based on lagThreshold.

Boundary rule: maxReplicaCount × container.concurrency ≤ topic.partitions. Pods beyond this ceiling join the group, receive zero partitions, and waste cloud spend.

Batch processing + DLQ error isolation

Use BatchListenerFailedException(message, cause, index) to pinpoint the exact failed record in a batch. Spring commits offsets for all records before that index, retries the failed one (with backoff), and routes it to the DLQ if retries expire — without reprocessing the rest of the batch.

8. KRaft (Kafka Raft Metadata Mode)

Why ZooKeeper was removed

ZooKeeper-based Kafka required a separate cluster, dual security models, and had a hard ceiling of ~200K partitions. A controller crash required reading and deserializing the entire cluster state from ZooKeeper — stalling the cluster for minutes. KRaft treats metadata as a native Kafka commit log, enabling sub-second failover and 1M+ partition clusters.

Node roles (process.roles)

• controller: Participates in Raft consensus, manages metadata log. Never handles user data.
• broker: Stores topic partitions, serves producers and consumers.
• broker,controller (combined): Development/small clusters only. Heavy broker I/O can starve the quorum.

The @metadata partition and Raft

All cluster mutations (topic creation, partition assignment, leader elections, ACLs) are written as events to the internal @metadata log and replicated via Raft. Standby controllers replicate in real time — on leader failure, the winner already has the full state in RAM. Zero state-loading delay.

Broker metadata consumption

Brokers are internal consumers of the @metadata log. They pull updates from the active controller leader asynchronously, maintaining a warm in-memory routing cache. Client metadata queries are served from local RAM — no controller round-trips required.

Quorum sizing rule

Fault tolerance F → use 2F+1 controller nodes (always odd)
3 controllers → tolerate 1 failure (majority = 2)
5 controllers → tolerate 2 failures (majority = 3)

Even-numbered quorums (e.g. 4) offer no additional fault tolerance over 3 nodes, only extra consensus latency. Always deploy controllers on dedicated hardware separate from data brokers.

ZooKeeper vs. KRaft at a glance

9. Interview Quick-Reference Card

Producer configs cheatsheet

Consumer troubleshooting matrix

One-liner definitions to memorize

• Offset: Monotonically increasing 64-bit integer, unique per partition. Partition-local — offset 100 on P0 has nothing to do with offset 100 on P1.
• ISR: In-Sync Replicas — followers keeping up with leader within replica.lag.time.max.ms.
• High Watermark: Highest offset replicated to all ISR members. Consumers can never read past it.
• Log Compaction: Background process that retains only the latest record per key. Tombstone (null value) = delete the key.
• Consumer Lag: HW − committed offset. The #1 Kafka health metric to monitor.
• Sticky Partitioner: Default for keyless messages. Fills one batch before rotating partitions. Better compression than round-robin.
• KRaft: Embedded Raft consensus replacing ZooKeeper. Metadata stored as @metadata commit log. Sub-second failover.