Kafka KRaft: Architecture, Consensus & Metadata Internals

For more than a decade, running Apache Kafka required operating two separate distributed systems: Kafka brokers for event data and an external Apache ZooKeeper ensemble for cluster metadata, controller elections, and topic configuration. Operating this dual-tier architecture introduced operational friction, security complexity, and severe partition scalability limits.

Introduced in KIP-500 and officially production-ready in modern Kafka releases, KRaft (Kafka Raft Metadata mode) replaces ZooKeeper entirely. By running an event-driven consensus engine natively inside Kafka, KRaft enables clusters to scale to millions of partitions, achieve sub-second controller failovers, and streamline cluster management into a single unified binary.

1. The End of ZooKeeper: Why Kafka Re-Engineered Consensus

To understand KRaft, one must first recognize the fundamental architectural bottlenecks of the legacy ZooKeeper model:

In the ZooKeeper architecture, cluster state was maintained as an external hierarchical tree of znodes. Exactly one broker was elected as the cluster Controller, responsible for reading metadata from ZooKeeper and propagating updates to all other brokers via synchronous RPCs. When partition counts grew past a few hundred thousand, this design degraded severely. If the active Controller broker crashed, the newly elected Controller had to read and deserialize the entire cluster state from ZooKeeper before resuming operations—a process that frequently stalled clusters for several minutes.

KRaft eliminates this bottleneck by embracing Kafka's own core philosophy: treating metadata as an immutable, event-driven commit log. Instead of synchronizing state with an external coordination service, Kafka stores its metadata in an internal replicated topic partition, keeping controllers in constant sync with zero state-loading delay.

2. KRaft Node Roles: Controllers, Brokers, and Combined Nodes

Under KRaft, every node in a Kafka cluster defines its operational identity through the process.roles configuration parameter:

Dedicated Controllers (process.roles=controller): These nodes form the Controller Quorum. They do not store user topic partitions or serve producer and consumer traffic. Instead, they participate in Raft consensus, append metadata changes to the metadata log, and manage partition leadership across the cluster.

Dedicated Brokers (process.roles=broker): These nodes act purely as data storage workers. They persist topic segment files, process produce and consume requests, manage local disk indexes, and pull metadata updates from the active controller.

Combined Nodes (process.roles=broker,controller): A single JVM process performs both duties simultaneously. While cost-effective for local development and staging environments, running combined nodes in high-throughput production clusters is discouraged because heavy message I/O and garbage collection pauses can starve the controller quorum of CPU cycles.

flowchart TB
    subgraph Quorum["KRaft Controller Quorum (Raft Consensus)"]
        direction LR
        C1["Controller 1\n(Active Leader)"]
        C2["Controller 2\n(Standby Voter)"]
        C3["Controller 3\n(Standby Voter)"]
        C1 <-->|"Raft Heartbeats & Log Sync"| C2
        C1 <-->|"Raft Heartbeats & Log Sync"| C3
    end

    subgraph BrokerPool["Broker Tier (Data Storage & Client Traffic)"]
        direction LR
        B1["Broker 101\n(Local Metadata Cache)"]
        B2["Broker 102\n(Local Metadata Cache)"]
        B3["Broker 103\n(Local Metadata Cache)"]
    end

    C1 -->|"Metadata Fetch RPC"| B1
    C1 -->|"Metadata Fetch RPC"| B2
    C1 -->|"Metadata Fetch RPC"| B3

3. The @metadata Topic & Raft Consensus Mechanics

At the center of KRaft is a specialized internal partition named @metadata. Unlike ordinary topic partitions that replicate using Kafka's traditional pull-based ISR model, the metadata partition is governed strictly by the Raft consensus algorithm.

Within the controller quorum, exactly one controller is elected as the active Leader, while the remaining quorum members act as Followers (Voters). Whenever an administrative change occurs—such as creating a topic, reassigning partition replicas, or electing a partition leader—the active Controller Leader constructs a metadata record, appends it locally to its @metadata log, and replicates it to the standby voter controllers.

Once a strict majority of quorum voters acknowledge the write, the metadata change is committed. Because standby controllers replicate every event in real time, if the active leader suffers a hardware failure, an election concludes instantly. The winning voter already has the full metadata log replicated in memory and can assume leadership in tens of milliseconds without reloading state.

4. How Brokers Consume Metadata: Event-Driven Pull

One of the most consequential architectural changes in KRaft is how regular data brokers receive cluster state updates. In ZooKeeper-based Kafka, the active controller pushed updates out to brokers using individual synchronous RPCs (such as LeaderAndIsrRequest and UpdateMetadataRequest), often creating network queuing bottlenecks during large cluster transitions.

In KRaft, metadata distribution is inverted: data brokers act as internal consumers of the metadata log. Each broker continuously issues asynchronous Fetch and FetchSnapshot requests to the active Controller Leader, pulling new metadata batches as they commit.

sequenceDiagram
    autonumber
    actor Admin as Admin Client / Producer
    participant Leader as Active Controller Leader
    participant Voter as Standby Controller Voter
    participant Broker as Data Broker Node

    Admin->>Leader: CreateTopic("payments", partitions=12)
    activate Leader
    Note over Leader: Encodes topic creation record
Appends to local @metadata log
    
    Leader->>Voter: ReplicateMetadataRecord(epoch=2, offset=450)
    activate Voter
    Voter-->>Leader: ReplicationACK(offset=450)
    deactivate Voter
    Note over Leader: Quorum majority confirmed
Metadata record committed
    Leader-->>Admin: TopicCreatedResponse(SUCCESS)
    deactivate Leader

    Broker->>Leader: FetchMetadata(last_offset=449)
    activate Broker
    activate Leader
    Leader-->>Broker: FetchResponse(records=[offset=450])
    deactivate Leader
    Note over Broker: Updates in-memory partition table
Ready to route client requests immediately
    deactivate Broker

This event-driven pull model ensures that brokers maintain warm, in-memory caches of the entire cluster routing map. When a producer or consumer queries a broker for topic metadata, the broker serves the response immediately from local RAM without contacting controllers.

5. ZooKeeper vs. KRaft: Architecture & Scale Comparison

The removal of ZooKeeper represents the largest architectural overhaul in Apache Kafka's history. The operational differences between the two modes reflect profound improvements in scalability, reliability, and deployment simplicity:

6. Production Quorum Sizing & Best Practices

Designing a production KRaft deployment requires strict adherence to consensus quorum sizing laws to ensure high availability during hardware outages:

The Quorum Majority Formula (2F + 1): Because Raft requires a strict majority to commit metadata and elect leaders, controller quorums must always consist of an odd number of voting members. A 3-node controller quorum can tolerate the loss of 1 node (F=1) while maintaining a voting majority of 2. A 5-node controller quorum tolerates 2 failures (F=2) with a majority of 3. Even-numbered quorums (such as 4 nodes) offer zero additional fault tolerance over 3 nodes while adding network latency to consensus votes.

Separation of Controller and Broker Hardware: In enterprise production environments, always deploy controllers on dedicated nodes separate from storage brokers. High-volume broker workloads experience heavy page cache churn, NVMe disk I/O, and periodic JVM garbage collection pauses that can trigger false heartbeats or destabilize the metadata quorum if shared on the same host.

Essential KRaft Configuration Keys:

process.roles defines whether the node acts as a broker, controller, or both. node.id establishes a unique integer identifier for the instance. controller.quorum.voters specifies the static list of voting controller endpoints in the format {id}@{host}:{port} (e.g., 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093). Finally, controller.listener.names identifies the secure, dedicated network listener reserved exclusively for inter-controller and broker-to-controller consensus traffic.