Part 4: Streaming, Messaging & Real-Time Systems (Questions 150–169)

Curated from high-throughput streaming engineering loops at Confluent (Apache Kafka), Uber, Netflix (Apache Flink), and Google, this guide covers 20 real-time messaging, event-time windowing, and streaming architecture interview questions. Below are the complete questions and detailed, human-written architectural answers, complete with schema trade-offs, internal mechanics, ASCII diagrams, and production code snippets.

Question 150: Describe Apache Kafka's core architecture (Brokers, Topics, Partitions, and Consumer Groups).

Kafka is a distributed append-only commit log:

• Topic: A logical category or stream to which records are published.
• Partition: Topics are split into ordered, immutable partitions distributed across brokers for horizontal scalability. Each message within a partition is assigned a sequential ID called an Offset.
• Broker: A Kafka server node managing partitions and handling read/write requests.
• Consumer Group: A pool of consumers that divide topic partitions among themselves. Each partition is consumed by exactly one consumer per consumer group. This enables parallel horizontal scaling.

graph LR
    subgraph KafkaCluster ["Kafka Topic: events (3 Partitions)"]
        P0[Partition 0]
        P1[Partition 1]
        P2[Partition 2]
    end

    subgraph ConsumerGroup ["Consumer Group A (3 Workers)"]
        C1[Consumer 1]
        C2[Consumer 2]
        C3[Consumer 3]
    end

    P0 --> C1
    P1 --> C2
    P2 --> C3

Question 151: What is the difference between At-Least-Once, At-Most-Once, and Exactly-Once delivery?

• At-Most-Once: Messages may be lost but are never duplicated. Offsets are committed before the message is processed. If the worker crashes mid-processing, the message is skipped.
• At-Least-Once: Messages are guaranteed never to be lost, but duplicates can occur. Offsets are committed after the message is processed. If the worker crashes after processing but before committing the offset, the replacement consumer re-processes the event.
• Exactly-Once Processing (EOS): Each message affects downstream state exactly once. Requires an Idempotent Producer, Kafka Transactional API, and an idempotent sink or two-phase commit downstream.

Question 152: How does Kafka determine which partition receives a message?

• Key Provided: Kafka hashes the key using MurmurHash2: partition = MurmurHash2(key) % total_partitions. All messages sharing the identical key (e.g., user_id = 42) are guaranteed to land on the same partition, ensuring strict FIFO ordering per entity.
• No Key (Null): Kafka uses the Sticky Partitioner strategy, batching messages together to fill a single partition before switching to the next, optimizing network throughput and compression.

Question 153: What is Consumer Lag and what are the top 3 ways to fix it?

Consumer Lag is the delta between the latest offset written by producers (Log End Offset) and the current offset committed by consumers. Rising lag means your downstream consumers cannot keep up with incoming volume.

Question 154: How does Kafka Log Compaction work and what is its primary use case?

Standard Kafka topics delete segments based on retention time (e.g., 7 days) or size. In a Log-Compacted Topic, Kafka retains at least the last known value for each message key forever. Cleaner threads scan historical log segments and purge obsolete records that have been superseded by newer updates for that key.

Use Case: Maintaining the latest state of database tables for Change Data Capture (CDC), or storing user profile caches that need to be replayed during system reboots.

Question 155: What is a Tombstone message in Kafka?

In a log-compacted topic, if you want to delete a key entirely, producing a message with that key and a null payload emits a Tombstone. The log cleaner retains the tombstone for a configurable period (delete.retention.ms) to give consumers time to see the deletion, after which it purges the key and tombstone from disk completely.

Question 156: Explain Event Time, Processing Time, and Ingestion Time in stream processing.

• Event Time: The exact time the event occurred on the client device or origin system (embedded in the message payload). Always use Event Time for accurate business analytics.
• Ingestion Time: The timestamp when the Kafka broker appended the message to its log.
• Processing Time: The local machine clock time of the streaming engine (Flink, Spark) when processing the record. Subject to machine clock skew and backpressure delays.

Question 157: What is Change Data Capture (CDC) and how does Debezium work?

CDC streams real-time row-level database changes (INSERT, UPDATE, DELETE) into downstream systems without polling tables. Debezium attaches directly to the database's internal transaction write-ahead log (e.g., Postgres WAL, MySQL Binlog). It decodes low-level binary mutation events, wraps them in structured JSON/Avro envelopes with before-and-after state, and publishes them into Kafka topics in real time with near-zero overhead on the source database.

Question 158: What makes a data pipeline Idempotent, and why is it crucial for backfills?

An operation is idempotent if executing it multiple times produces the exact identical outcome as running it once: f(f(x)) = f(x).

Why it matters: Distributed pipelines fail frequently due to spot instance termination, network timeouts, or bad deployments. If your pipeline isn't idempotent, retrying a failed batch inserts duplicate records.

Question 159: What causes a Kafka Consumer Group Rebalance, and how do you prevent the "Rebalance Storm"?

A rebalance occurs when consumers join, leave, or are deemed dead, forcing partition assignments to redistribute.

The Trap: If a consumer takes too long to process a heavy batch of records, it exceeds max.poll.interval.ms before invoking poll() again. The Kafka Broker assumes the consumer is dead, kicks it out of the group, and triggers a rebalance. All other consumers freeze while rebalancing, causing a cascading storm!

Solution: Lower max.poll.records, increase max.poll.interval.ms, or adopt Cooperative Sticky Assignor which reassigns only necessary partitions without pausing the entire group.

Question 160: Compare Apache Flink vs. Apache Spark Structured Streaming.

• Apache Flink: True event-driven, record-by-record streaming engine. Sub-millisecond latency. Superior state management (RocksDB state backend) and out-of-the-box complex event processing (CEP).
• Spark Structured Streaming: Micro-batch architecture by default (batches events every 100ms–1s). Higher throughput for batch/stream unification, easier to operate for teams already skilled in Spark, but slightly higher latency than native Flink.

Question 161: What is Backpressure in streaming systems, and how do engines respond?

Backpressure occurs when an upstream producer sends data faster than a downstream operator can process. Without backpressure, queues fill up and the system crashes with OOM. Modern engines (Flink, Spark) signal upstream operators to throttle reading from source message brokers until local queues clear.

Question 162: What is Confluent Schema Registry and why is it used in Kafka?

Kafka is payload-agnostic; it simply moves bytes. If a producer changes a JSON payload from user_id: int to user_id: string, downstream consumers crash.

Schema Registry stores versioned Avro/Protobuf/JSON schemas:

• Producers register schemas and attach only a 4-byte Schema ID in the message wire protocol instead of embedding the entire schema in every message.
• It enforces schema evolution rules (Backward, Forward, Full compatibility) and rejects messages that would break downstream consumers.

Question 163: How do Kafka Transactions achieve Exactly-Once Semantics across read-process-write loops?

In a stream processing application reading from Topic A and writing to Topic B:

1. The producer registers a transactional.id with the Transaction Coordinator broker.
2. The consumer offsets for Topic A and the output records for Topic B are written within a single atomic transaction.
3. A two-phase commit writes a commit marker to the transaction log. Downstream consumers configured with isolation.level = read_committed will only see records whose transaction markers are fully committed.

Question 164: Contrast Lambda Architecture vs. Kappa Architecture.

• Lambda Architecture: Maintains two distinct paths: a fast Speed Layer (streaming) for low-latency dashboards, and a Slow Batch Layer (Spark/Hadoop) for accurate historical reconciliation. Flaw: Requires writing, testing, and debugging business logic twice across two different codebases.
• Kappa Architecture: Eliminates the batch layer completely! Uses a single stream-processing engine (e.g., Flink/Kafka) for both real-time processing and historical reprocessing by simply resetting consumer offsets to 0.

Question 165: Explain Tumbling, Sliding, and Session Windows.

• Tumbling Window: Fixed size, non-overlapping (e.g., 00:00–00:05, 00:05–00:10). Every event belongs to exactly one window.
• Sliding (Hopping) Window: Fixed size, overlapping (e.g., a 10-minute window that evaluates every 1 minute). Events can belong to multiple windows.
• Session Window: Dynamic window based on inactivity gap (e.g., group all user clicks until there is a 30-minute period of inactivity, marking the end of a session).

Question 166: What is a Dead Letter Queue (DLQ) pattern and how is it implemented in streaming pipelines?

When a consumer encounters a malformed or corrupted message ("poison pill") that fails deserialization, halting the consumer halts the entire pipeline.

DLQ Pattern: The consumer catches the unparseable exception, writes the raw bad message along with error stack trace metadata to a dedicated dead_letter_topic, and commits the offset to keep the primary pipeline flowing. Engineers can inspect, fix, and replay the DLQ topic later.

Question 167: What is Zero-Copy data transfer in Kafka and why is it so fast?

Traditional data transfer requires 4 context switches and copying data between kernel and user space: Disk → OS Cache → Application Buffer → Socket Buffer → NIC. Kafka uses the Linux sendfile() system call to transfer bytes directly from OS Page Cache to the Network Interface Card (NIC) buffer, completely bypassing user space. This cuts CPU cycles and memory bus copies in half.

Question 168: How does Apache Flink achieve Fault Tolerance via Asynchronous Barrier Snapshotting (ABS)?

Flink periodically injects lightweight Checkpoint Barriers into the stream among regular records. As barriers flow through operators, each operator freezes its local state (in RocksDB or RAM) asynchronously to persistent storage (S3). If an operator fails, all operators rollback state to the last aligned barrier snapshot, ensuring fault tolerance without halting the pipeline.

Question 169: How do you enrich a high-throughput stream with slowly changing reference data?

Querying an external database (PostgreSQL/Redis) for every single stream event introduces catastrophic network latency (e.g., 5ms * 50,000 events/sec = crash).