@ 9f9fda7f:e5e82a43

Apache Kafka Architecture

Apache Kafka Architecture

Apache Kafka is a powerful distributed stream processing platform originally developed by LinkedIn, written in Scala and Java. In this article, I walk through the foundational concepts, design and architecture, concluding with hands-on coding examples using Node.js and the KafkaJS library.

Introduction to Kafka

Kafka enables the processing of real-time data streams in a distributed and scalable manner. It’s widely used in systems requiring reliable communication between components, especially for event-driven architectures and microservices.

However, these words sounds very markety and I like to deconstruct things to its basic first principles so that is what I’m going to do. This guide outlines the essential components, explains the core concepts like brokers, producers, consumers, and topics, and dives into more complex abstractions like partitions, consumer groups, and Kafka’s distributed nature.

Kafka Core Components

Here we discuss the fundamental components of Kafka.

Topics and Messages

Messages in Kafka are organized into logical categories called topics. Each topic is append-only and immutable (ie you cannot go back and edit a message). New data is added sequentially at the end. Disks love that.

There are two topics, one called users with two messages (John and Ed) and a Jobs with two messages Req1 and Req2.

Kafka Broker (Server)

The central server that receives and stores messages sent by producers and allow consumers to read the messages. Clients communicate with brokers via TCP, typically on port 9092.

Kafka broker hosts the topics and messages.

Producers and Consumers:

Producers connects to a broker, sends messages to the Kafka broker by specifying the topic.

Publisher pushes messages to the broker for a topic, messages go to the end.

Consumers connects to a broker, polls messages from a topic for a given broker for processing.

Customer polls messages based on a position

Partitions

I always say the best way to work with billions of rows is to avoid working with billions of row. Kafka is no different.

The topics in Kafka can be divided into partitions to allow data distribution and scalability. Each partition acts as a log that consumers read sequentially. Messages within partitions are identified by their offset, which is the message’s position in the log.

Partitioning

Partitioning enables parallel processing, you can have two consumers each processing a partition in parallel. While this allows scaling and improve reading throughput, it introduces complexity as producers and consumers must be aware of partitioning logic. A topic may have multiple partitions based on partition keys (e.g., A–M in partition 0, N–Z in partition 1).

Pub/Sub vs Queue Model

Kafka supports both pub/sub and queue-based messaging patterns:

• Queue: The topic can be treated as a queue, ie any message published to the queue will be consumed only once and popped from the queue. No two consumers can get the same message.
• Pub/Sub: The topic can be in publish-subscribe entity, ie any message published to the topic can be consumed by different consumers.

Kafka achieves this dual functionality using consumer groups. Each consumer in a group is assigned specific partitions, ensuring messages are processed only once per group.

To simulate a queue , all consumers are placed in the same consumer group (they all have the same group name). Whereas to simulate pub/sub mode you would place all consumers in unique group names.

Consumer Groups and Rebalancing

Each partition in a topic is assigned one consumer in a group. When multiple consumers join the same group, Kafka rebalances partitions among them. Each partition is consumed by only one consumer in a group. This allows horizontal scaling while ensuring order within each partition is maintained. Will demonstrate that in the code section.

Distributed Kafka Clusters and Zookeeper

Kafka operates as a distributed system with multiple brokers. Each broker can be a leader or follower for different partitions. The leader handles reads and writes, while followers replicate data.

Kafka originally relied on Zookeeper to manage broker coordination, metadata, and leader election. Zookeeper tracks which broker is the leader of which partition. However, Zookeeper introduces complexity and is being phased out in newer Kafka versions. Kafka 4.0 no longer has Zookeeper.

On Long Polling

You may ask, how does the consumer get the data the moment it is published to the topic/partition? I encourage you to first think of how could this be implemented.

One approach is push, ie the broker pushes the message to the consumer TCP connection. This approach is real-time, but overwhelms the consumer, as it needs to process the message.

Another approach is polling, ie the consumer every X milliseconds asks the broker, is there anything for me and the broker replies, yes here you go, or no nothing, this approach does allow the consumer to read at its own pace but may introduce severe network chatter, which lead to network bandwidth, it also may causes consumer to miss the real-time message, since the message may have came in to the broker between polls.

The final approach which Kafka went with is long polling which I talked extensively on my backend course. The consumer connects to the broker and sends a normal poll, if there is a message the broker replies back, if there isn’t the broker doesn’t respond with “no”, but instead just wait little bit more (configurable), this increases the chances that messages may arrive during this wait. Once we get a message the broker writes the response. Meanwhile the consumer feels like its poll request just took little longer.

Kafka, Docker and Node Example

We will use docker to install Kafka broker and Zookeeper single mode cluster. Then create a topic called “Users” with two partitions, produce some messages on both partitions and have consumers read from the partitions.

My docker host IP is 192.168.7.179, replace it with your own docker host IP.

You will need to have docker and Node installed to proceed

First we clone the repo and change IP address to your docker host.

```javascript

clone the repo

git clone https://github.com/hnasr/javascript_playground.git

change to the kafka directory

cd javascript_playground/kafka

edit docker-compose.yml, producer.js, topics.js and consumer.js change the 192.168.7.179 with your docker host ip.

cat starthere.txt ```

Spin up Zookeeper and Kafka containers using Docker commands.

```javascript services: zookeeper: image: zookeeper hostname: zookeeper ports: - 2181:2181 volumes: - zookeeper_data:/data - zookeeper_datalog:/datalog

kafka: image: ches/kafka restart: always hostname: localhost ports: - "9092:9092" environment: KAFKA_ADVERTISED_HOST_NAME: 192.168.7.179 ZOOKEEPER_IP: 192.168.7.179 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 ALLOW_PLAINTEXT_LISTENER: 1 depends_on: - zookeeper volumes: - kafka_data:/kafka

volumes: zookeeper_data: zookeeper_datalog: kafka_data: ```

javascript docker-compose up

Create a topic called MyUsers with two partitions, we will use kafkajs module to connect to the kafka broker.

```javascript //const Kafka = require("kafkajs").Kafka const {Kafka} = require("kafkajs")

run(); async function run(){ try { const kafka = new Kafka({ "clientId": "myapp", "brokers" :["192.168.7.179:9092"] })

    const admin = kafka.admin();
    console.log("Connecting.....")
    await admin.connect()
    console.log("Connected!")
    //A-M, N-Z
    await admin.createTopics({
        "topics": [{
            "topic" : "MyUsers",
            "numPartitions": 2
        }]
    })
    console.log("Created Successfully!")
    await admin.disconnect();
}
catch(ex)
{
    console.error(`Something bad happened ${ex}`)
}
finally{
    process.exit(0);
}

} ```

javascript node topics.js

Produce some messages with a producer, the messages are user names. I have the messages starting with letters

```javascript //const Kafka = require("kafkajs").Kafka const {Kafka} = require("kafkajs") const msg = process.argv[2]; run(); async function run(){ try { const kafka = new Kafka({ "clientId": "myapp", "brokers" :["192.168.7.179:9092"] })

    const producer = kafka.producer();
    console.log("Connecting.....")
    await producer.connect()
    console.log("Connected!")
    //A-M 0 , N-Z 1 
    const partition = msg[0] < "N" ? 0 : 1;
    const result =  await producer.send({
        "topic": "Users",
        "messages": [
            {
                "value": msg,
                "partition": partition
            }
        ]
    })

    console.log(`Send Successfully! ${JSON.stringify(result)}`)
    await producer.disconnect();
}
catch(ex)
{
    console.error(`Something bad happened ${ex}`)
}
finally{
    process.exit(0);
}

} ```

javascript node producer.js Adam node producer.js Zack

javascript HusseinMac:kafka HusseinNasser$ node producer.js Adam Connecting..... Connected! Send Successfully! [{"topicName":"MyUsers","partition":0,"errorCode":0,"offset":"0","timestamp":"-1"}] HusseinMac:kafka HusseinNasser$ node producer.js Zack Connecting..... Connected! Send Successfully! [{"topicName":"MyUsers","partition":1,"errorCode":0,"offset":"0","timestamp":"-1"}]

Implement a consumer that subscribes to the topic and handles messages with long polling.

Note that I hard coded a consumer group name “test” in the consumer.js file. Running the consumer.js file will create a new consumer on the MyUsers topic and it will be responsible for both partitions. You can note this from the response"memberAssignment”:{“MyUsers”:[0,1]}

javascript HusseinMac:kafka HusseinNasser$ node consumer.js Connecting..... Connected! {"level":"INFO","timestamp":"2025-04-13T14:29:36.102Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"test"} {"level":"INFO","timestamp":"2025-04-13T14:29:58.411Z","logger":"kafkajs","message":"[Runner] Consumer has joined the group","groupId":"test","memberId":"myapp-901679c6-034c-4efd-bc4d-cb5fc5bc9e6e","leaderId":"myapp-901679c6-034c-4efd-bc4d-cb5fc5bc9e6e","isLeader":true,"memberAssignment":{"MyUsers":[0,1]},"groupProtocol":"RoundRobinAssigner","duration":22307} RVD Msg Adam on partition 0 RVD Msg Zack on partition 1

Now assume we run another consumer, which will also have the same group. Kafka will rebalance the group and assign a partition for each consumer so they parallel process.

Note the memberAssignment property for each consumer.

```javascript HusseinMac:kafka HusseinNasser$ node consumer.js (consumer 1) {"level":"ERROR","timestamp":"2025-04-13T14:31:09.360Z","logger":"kafkajs","message":"[Connection] Response Heartbeat(key: 12, version: 0)","broker":"192.168.7.179:9092","clientId":"myapp","error":"The group is rebalancing, so a rejoin is needed","correlationId":39,"size":6} {"level":"ERROR","timestamp":"2025-04-13T14:31:09.361Z","logger":"kafkajs","message":"[Runner] The group is rebalancing, re-joining","groupId":"test","memberId":"myapp-901679c6-034c-4efd-bc4d-cb5fc5bc9e6e","error":"The group is rebalancing, so a rejoin is needed","retryCount":0,"retryTime":308} {"level":"INFO","timestamp":"2025-04-13T14:31:09.382Z","logger":"kafkajs","message":"[Runner] Consumer has joined the group","groupId":"test","memberId":"myapp-901679c6-034c-4efd-bc4d-cb5fc5bc9e6e","leaderId":"myapp-901679c6-034c-4efd-bc4d-cb5fc5bc9e6e","isLeader":true,"memberAssignment":{"MyUsers":[1]},"groupProtocol":"RoundRobinAssigner","duration":21}

HusseinMac:kafka HusseinNasser$ node consumer.js (consumer 2) Connecting..... Connected! {"level":"INFO","timestamp":"2025-04-13T14:31:07.599Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"test"} {"level":"INFO","timestamp":"2025-04-13T14:31:09.383Z","logger":"kafkajs","message":"[Runner] Consumer has joined the group","groupId":"test","memberId":"myapp-6c861d03-879d-4b7b-81a1-1e2cd1c15c83","leaderId":"myapp-901679c6-034c-4efd-bc4d-cb5fc5bc9e6e","isLeader":false,"memberAssignment":{"MyUsers":[0]},"groupProtocol":"RoundRobinAssigner","duration":1783} ```

Let us publish one message for each partition and see what happens.

javascript node producer.js Adamy node producer.js Zackery

```javascript (consumer 1) RVD Msg Zackery on partition 1 (consumer 2) RVD Msg Adamy on partition 0

```

What happens if more consumers than partitions are available? As per testing, new consumers won’t get any partitions assigned. But a consumer group rebalance may reassign partitions.

Kafka Pros and Cons

By understanding the architecture and design of Apache Kafka we know the fact it is performs well for writes (producing messages), because of the append-only nature. And having the consumer aware of the position to where to read from makes reads also fast. Kafka scales well with partitioning and handle fault tolerant with the distributed nature.

However, no software is free of limitations, with all these features added to Kafka, complexity is unavoidable. Producers needs to be aware of the partitions which introduces complexity in producing messages. The need for distributed architecture, replication introduces depedancies on an external component (Zookeeper) which adds more complexity and even latency. All of this makes Kafka difficult to install, maintain and manage.

Note that Kafka 4.0 that was release in March 2025 has completely removed Zookeeper in favor of a raft based replication between brokers.

Conclusion

Now that you know the basics of Apache Kafka, you will understand its power and its limitations. That is the most important thing. Anything on top of Kafka uses these fundamentals.

If you enjoyed this article, check out my backend and database courses.

https://stas.layer3.press/articles/c11f511b-29f5-4612-a308-004c44183c7b