Apache Kafka Consumer Example from CLI
Kafka Console Consumer Tool
We can access the Kafka consumer CLI tool using the Kafka-console-consumer.sh utility. This tool is part of the Apache Kafka distribution and is implemented in Java. The Kafka Consumer API provides a high-level, abstracted interface to consume the records from a Kafka cluster.
Using the CLI interface, the tool can perform as a consumer client which allows us to connect a specified Kafka broker and fetch the messages from the specified topics.
It then displays the messages from the topic in the terminal output, providing an easy way to view and inspect the data that are produced by producers.
This tutorial explores how to set up a basic Kafka consumer client using the Kafka Consumer CLI interface.
Create a Sample Topic in Kafka
Let us start by creating a sample topic on which we subscribe and read the messages. We use the kafka-topics.sh utility.
For example, suppose we want to create a new topic called “coordinates”. We can run the command as shown:
In the given example, we use the –create option to set up a new topic called “coordinates”. We then specify the replication factor of 3. This means that each topic partition is replicated across 3 Kafka brokers.
Finally, we specify the number of partitions as 3. This means that the topic is divided into three partitions.
Produce Messages in Kafka
Once we created the topic, we can write to it using the kafka-console-producer command. An example is as follows:
> {"latitude": 37.7749, "longitude": -122.4194}
> {"latitude": 40.7128, "longitude": -74.0060}
> {"latitude": 51.5074, "longitude": -0.1278}
> {"latitude": 35.6895, "longitude": 139.6917}
> {"latitude": 55.7558, "longitude": 37.6173}
> {"latitude": -33.8651, "longitude": 151.2093}
> {"latitude": 48.8566, "longitude": 2.3522}
> {"latitude": 37.5665, "longitude": 126.9780}
> {"latitude": 35.6762, "longitude": 139.6503}
> {"latitude": 43.6532, "longitude": -79.3832}
> EOF
In the previous command, we use the kafka-console-producer.sh utility to produce the messages to the “coordinates” topic. The –topic parameter lets us specify which topic we wish to write.
Consume Messages in Kafka
As you can guess, we can consume the messages from the “coordinates” topic using the kafka-console-consumer application as shown in the following:
In this case, the command should read the messages from the coordinates from the beginning as determined by the –from-beginning parameter.
Example Output:
Conclusion
You now learned how to use the Kafka CLI tools to create a new topic, produce messages, subscribe to a Kafka topic, and read messages from it.
Source: linuxhint.com