| by Arround The Web | No comments

Apache Kafka Consumer Example in NodeJS

In the modern world, data is one of the most essential and valuable tools for any application. However, although the traditional databases are powerful in handling static information, they fall short and cannot take such data regarding near real-time data flow.

This is where the tools such as Apache Kafka comes into play. Apache Kafka, commonly known as Kafka, is a free and open-source distributed event streaming platform. Kafka uses the pub-sub model that allows the publishers to write the events. At the same time, the subscribers can read the messages that are written by the publishers.

Although Kafka supports a wide range of programming languages, we will learn how to create a basic Kafka consumer application using Node.JS in this tutorial.

Since its launch, Node.JS quickly became one of the most widely adopted JavaScript runtime environments. Thanks to the external packages, it has continued to grow and power the small to large enterprise applications.


For you to follow along with this tutorial and ensure a maximum compatibility, we recommend the following:

  1. Installed latest version of Apache Kafka on your local machine
  2. Node.JS version 18.50
  3. Basic JavaScript development knowledge

Get your IDEs ready, and let us dive in!

Project Setup

Start by creating a directory to store the source code for your project.

$ mkdir kafka-node

Navigate into the created directory:

$ cd kafka-node

Install the Required Dependencies

The next step is to install the required libraries. For this tutorial, we use the node-rdkafka library.

The node-rdkafka is a free, open-source, high-performance Node.JS client library to interact with Apache Kafka. The library leverages the power of the librdkafka C/C++ library. This allows us to build the efficient, robust consumer and producer applications in Node.JS with low latency and high throughput from the librdkafka implementation.

To install this package, run the npm command as shown in the following:

$ npm install node-rdkafka

Initialize a New Kafka Topic

The next step is to create a new Kafka topic to which we read and write the messages. For this, we can use the Kafka CLI tools as shown in the following command:

$ ./bin/kafka-topics.sh --create --topic users --bootstrap-server localho
st:9092 --replication-factor 1 --partitions 1

The previous command should create a topic called “users” with a replication factor of 1 and one partitions.

Setup the Kafka Producer

In the kafka-node directory, create a file to store the source code for the producer application.

$ touch producer.js

Edit the file and add the code as shown in the following:

const kafka = require('node-rdkafka');

// Kafka broker config

const kafkaConfig = {

'metadata.broker.list': 'localhost:9092',


// Create a Kafka producer instance

const producer = new kafka.Producer(kafkaConfig);

// Connect the producer to the broker


// Wait for the 'ready' event to be emitted before sending messages

producer.on('ready', () => {

console.log('Kafka producer is ready');

// Send 5 messages to the 'users' topic

for (let i = 1; i <= 5; i++) {

const message = `User ${i}`;

producer.produce('users', null, Buffer.from(message));

console.log(`Sent message: ${message}`);


// Disconnect the producer from the broker after sending messages



// Handle errors

producer.on('error', (err) => {

console.error('Error in Kafka producer:', err);


Setup the Kafka Consumer

Next, create a file to store the source code for the consumer application.

$ touch consumer.js

Add the source code as follows:

const kafka = require('node-rdkafka');

// Kafka broker configuration

const kafkaConfig = {

'metadata.broker.list': 'localhost:9092',

'group.id': 'node-app',


// Create a Kafka consumer instance

const consumer = new kafka.KafkaConsumer(kafkaConfig);

// Subscribe to the 'users' topic


// Consume messages from the 'users' topic

consumer.on('data', (message) => {

console.log(`Received message: ${message.value.toString()}`);


// Connect the consumer to the broker


// Handle errors

consumer.on('error', (err) => {

console.error('Error in Kafka consumer:', err);


// on SIGINT

process.on('SIGINT', () => {

consumer.disconnect(() => {

console.log('Kafka consumer disconnected');




Save the source code and use the node.js to run both the producer and consumer applications.

$ node producer.js


Kafka producer is ready

Sent message: User 1

Sent message: User 2

Sent message: User 3

Sent message: User 4

Sent message: User 5

To read the sent messages, run the consumer application:

$ node consumer.js


User 1

User 2

User 3

User 4

User 5


This article teaches us how to set up a basic consumer and producer application for Apache Kafka using Node.JS. Feel free to explore the node-rdkafka documentation to learn more about how you can expand your application.

Share Button

Source: linuxhint.com

Leave a Reply