| by Arround The Web | No comments

Apache Kafka Consumer Example from CLI

The Kafka Consumer CLI tool is a command-line interface that allows us to subscribe to one or more Kafka topics and consume the messages from them. It is a simplistic but powerful command-line utility that comes in handy when debugging or testing an application for development purposes.

Kafka Console Consumer Tool

We can access the Kafka consumer CLI tool using the Kafka-console-consumer.sh utility. This tool is part of the Apache Kafka distribution and is implemented in Java. The Kafka Consumer API provides a high-level, abstracted interface to consume the records from a Kafka cluster.

Using the CLI interface, the tool can perform as a consumer client which allows us to connect a specified Kafka broker and fetch the messages from the specified topics.

It then displays the messages from the topic in the terminal output, providing an easy way to view and inspect the data that are produced by producers.

This tutorial explores how to set up a basic Kafka consumer client using the Kafka Consumer CLI interface.

Create a Sample Topic in Kafka

Let us start by creating a sample topic on which we subscribe and read the messages. We use the kafka-topics.sh utility.

For example, suppose we want to create a new topic called “coordinates”. We can run the command as shown:

./bin/kafka-topics.sh --create --topic coordinates --replication-factor 3 --partitions 3 --bootstrap-server localhost:9092

In the given example, we use the –create option to set up a new topic called “coordinates”. We then specify the replication factor of 3. This means that each topic partition is replicated across 3 Kafka brokers.

Finally, we specify the number of partitions as 3. This means that the topic is divided into three partitions.

Produce Messages in Kafka

Once we created the topic, we can write to it using the kafka-console-producer command. An example is as follows:

./bin/kafka-console-producer.sh --topic coordinates --bootstrap-server localhost:9092 << EOF
> {"latitude": 37.7749, "longitude": -122.4194}
> {"latitude": 40.7128, "longitude": -74.0060}
> {"latitude": 51.5074, "longitude": -0.1278}
> {"latitude": 35.6895, "longitude": 139.6917}
> {"latitude": 55.7558, "longitude": 37.6173}
> {"latitude": -33.8651, "longitude": 151.2093}
> {"latitude": 48.8566, "longitude": 2.3522}
> {"latitude": 37.5665, "longitude": 126.9780}
> {"latitude": 35.6762, "longitude": 139.6503}
> {"latitude": 43.6532, "longitude": -79.3832}

In the previous command, we use the kafka-console-producer.sh utility to produce the messages to the “coordinates” topic. The –topic parameter lets us specify which topic we wish to write.

Consume Messages in Kafka

As you can guess, we can consume the messages from the “coordinates” topic using the kafka-console-consumer application as shown in the following:

./bin/kafka-console-consumer.sh --topic coordinates --bootstrap-server localhost:9092 --from-beginning

In this case, the command should read the messages from the coordinates from the beginning as determined by the –from-beginning parameter.

Example Output:


You now learned how to use the Kafka CLI tools to create a new topic, produce messages, subscribe to a Kafka topic, and read messages from it.

Share Button

Source: linuxhint.com

Leave a Reply